Sunday, July 5, 2026

High performance computing - A self-scheduling algorithm

HPC for a data science and cloud project was set up - Self-Scheduling Algorithm with Parallel programming

Abstract

It has been a while since I last posted in this forum. Today, I prepared a schedule related to High-Performance Computing (HPC) and reviewed a recent project I completed for an HPC and Data Science course. This project focused on implementing HPC tools to address scheduling problems in cloud environments. I discovered that similar scheduling issues exist in battery pack management, as my co-project team member discussed at the time.

Currently, I have a comprehensive understanding of how to manage scheduling in cloud projects, especially for time series-dependent workloads. The study revolves around analyzing such a spiky workload using mpirun. We introduced a transformation step to the training dataset to normalize this spiky load. For the self-scheduling algorithm to effectively handle such a load, we monitored the mpithreads to ensure smooth resource usage. 

Introduction

Applying a self-scheduling algorithm to solve real-world applications as an objective, the inputs considered include a square matrix and constraints, along with the system configuration for hybrid setups. The output was an optimized vector to demonstrate self-scheduling options

3

Literature review


The self-scheduling algorithm can be applied for real-time analysis in data science, specifically for:
    1. Scheduling charge and discharge cycles of battery cells within a battery pack.
    2. Managing resources such as memory, processors, and storage for cloud elasticity.
    3. Introducing a novel programming technique determined during the project, utilizing the MPI (Message Passing Interface) package.
IBM_torc_py Architecture

All remote operations are conducted asynchronously through a dedicated server thread. This thread is responsible for the following tasks:

  • Inserting incoming tasks into the local queue of the process
  • Receiving completed tasks along with their results
  • Handling task-stealing requests

The internal architecture of torcpy is illustrated in the accompanying figure. As a result, tasks (also known as features) can be created and finalized using the `submit` and `wait` calls. For more details, refer to the link https://github.com/IBM/torc_py

Pseudo-code

Package steps based on torc

  1. Init

# torcpy execution starts, and MPI initialization happens

  1. Submit

# Switching to master-worker

>>Assign the calculations to workers via Submit

work inp=8.000, out=64.000 ...on node 0 worker 0 thread 139725221726080

  1. Receive, create, execute

    1. Received: 8.0^2=64.000

    2. Elapsed time for col1 =47.07 s

    3. Elapsed time for col2 =48.07 s

    4. TORCPY: node[0]: created=94, executed=94


Self-Scheduler wrapper

  • Init

  • Submit 

    • Work x^2

    • T x T Transpose

  • Receive, create, & Execute

    • node_id(): return the rank of the calling MPI process

    • worker_id(): return the global worker thread ID 

Methodology

Maximize performance with Matrix-Vector Multiplication that enhances vector scheduling by leveraging MPI and OpenMP. Start with initial MPI optimization, then unlock even more potential through coarse optimization using OpenMP. #MatrixMultiplication #MPI #OpenMP #Optimization #TechTips




HTTP request dataset

 

No of nodes

No of workers

Elapsed time in seconds

No of Records

1

2

15/0.0

30/7

2

2

8/0.0

30/7

3

2

5/0.0

30/7

2

1

15/1

30/7

6

1

5/1

30/7

1

1

30/1

30/7


Sample Output 

TORCPY: main starts
work inp=11.000, out=121.000 ...on node 0 worker 0 thread 140230837819264
work inp=16.000, out=256.000 ...on node 0 worker 1 thread 140229995116288
work inp=3.000, out=9.000 ...on node 0 worker 3 thread 140229978330880
ork inp=3.000, out=9.000 ...on node 1 worker 6 thread 140336443447040
work inp=1.000, out=1.000 ...on node 1 worker 5 thread 140337286150016
work inp=4.000, out=16.000 ...on node 1 worker 7 thread 140336435054336
work inp=5.000, out=25.000 ...on node 1 worker 8 thread 140336426661632
work inp=9.000, out=81.000 ...on node 1 worker 9 thread 140336418268928


Elapsed time=3.05 s

TORCPY: node[0]: created=30, executed=15

TORCPY: node[1]: created=0, executed=15


 

No of nodes

No of workers

Elapsed time in seconds

Rows/no of data records

1

1

47

94

1

2

15

30


Result discussion
The empirical evaluation of our self-scheduling algorithm demonstrates significant performance gains when leveraging hybrid parallel programming. By systematically scaling the architecture from 1 to 6 nodes and adjusting worker thread configurations, we observed a substantial reduction in overhead and processing times. For instance, in our HTTP request dataset benchmark, increasing the node count effectively minimized the elapsed runtime from 30 seconds down to just 5 seconds for a standard batch of 30 records. Furthermore, integrating the torc_py framework allowed for seamless, asynchronous master-worker task distribution. As highlighted by our node execution logs, tasks were efficiently balanced across available hardware threads (e.g., node[0] and node[1] evenly splitting 30 executed tasks), mitigating the spiky resource usage typically associated with heavy time-series workloads. These optimization vectors prove that the self-scheduling wrapper successfully balances computation and communication overhead. Ultimately, this framework provides a highly viable, real-time analysis solution for data science applications—ranging from maintaining cloud elasticity during unpredictable demand spikes to managing critical charge/discharge cycles within battery packs.
Future work
Sample Output for the Battery pack dataset (94/30 records) - with required training data shall be implemented for differing thread tests.

Conclusion
In conclusion, utilizing a self-scheduling algorithm can effectively address real-world applications by optimizing the processing of a square matrix while adhering to specified constraints, including system configurations for hybrid systems. The result of this approach will be a refined and optimized vector, ultimately enhancing the efficiency and performance of the system.

Labels: ,

Saturday, February 14, 2026

The Quantum Challenge: Manufacturing the Synthetic Brain

 The Quantum Challenge: Manufacturing the Synthetic Brain

For decades, the goal of creating a "synthetic brain" relied on traditional silicon transistors to mimic neurons and synapses. However, the sheer scale of the human brain—100 billion neurons and nearly a quadrillion synapses—would require massive server rooms and immense power if built with current technology.
A new frontier, Quantum Materials, promises to compress this architecture into a device the size of a human brain using only 20 Watts of power. Yet, moving from laboratory breakthroughs to mass manufacturing presents fundamental challenges that could redefine the future of industry.
1. Replicating Biological "Non-Locality"
One of the most difficult brain functions to manufacture is non-locality—the brain's ability for a stimulus in one area to affect non-neighboring neurons.
  • The Challenge: In traditional electronics, signals follow rigid, local paths through physical wires.
  • The Quantum Solution: Researchers have discovered that quantum materials like transition metal oxides exhibit "correlated behavior," where electrons interact collectively across distances.
  • Manufacturing Hurdle: Fabricating these materials so they maintain this delicate collective state consistently across billions of artificial synapses is exponentially more complex than printing standard silicon circuits.
2. Manufacturing "Frustrated" Lattices
To mimic the brain's ability to learn and forget, engineers are turning to "frustrated" magnetic states in quantum materials.
  • The Goal: Use triangular arrangements of lanthanide elements to create a state of "intrinsic quantum disorder" that can store memory like a synapse.
  • The Hurdle: This requires ultra-high crystal purity and atomic-scale precision during fabrication. Even a single misplaced atom can disrupt the quantum state, making large-scale manufacturing highly prone to defects.
3. The Temperature and Coherence Paradox
While some neuromorphic (brain-like) systems can work at room temperature, many high-performance quantum components require extreme conditions.
  • The Issue: Superconducting materials often need temperatures near absolute zero to function.
  • The Manufacturing Conflict: Building a portable "synthetic brain" that requires a massive cooling infrastructure is a logistical nightmare. Manufacturers are currently racing to develop materials like germanium-gallium alloys that can bridge the gap between quantum superconducting phases and standard room-temperature semiconductors.
4. Moving Beyond the "Edisonian" Approach
The traditional trial-and-error method of manufacturing (the Edisonian approach) is no longer feasible due to the complexity of quantum interactions.
  • The Shift: Future factories will likely need to integrate AI-driven materials design and first-principles calculations to predict how these materials will behave before a single chip is produced.
  • Integration: Companies like Samsung are already exploring how to integrate these quantum materials with existing CMOS manufacturing processes to create hybrid systems.
The Verdict
Quantum materials offer the only viable path to a true "brain-on-a-chip," but they force us to abandon 50 years of traditional semiconductor logic. The challenge is no longer just about making transistors smaller; it is about manufacturing emergence—the ability for a material to think, learn, and adapt by changing its own physical state.
Bibliography
  • Alexeev, Y., et al. (2021). "Quantum Computer Systems for Scientific Discovery." PRX Quantum. This source discusses the co-design of quantum systems and their applications, including the need for interdisciplinary approaches to overcome fabrication hurdles.
  • Basov, D. N., et al. (2017). "Towards properties and applications of quantum materials." Nature Materials. Provides the foundation for understanding how lattice, charge, and orbital degrees of freedom create the electronic states used in artificial synapses.
  • de Leon, N. P., et al. (2021). "Materials challenges and opportunities for quantum computing hardware." Science. A key reference for the manufacturing difficulties regarding material heterogeneity and the impact of impurities on quantum coherence.
  • Dai, S., et al. (2024). "Exploring quantum materials and applications: a review." Journal of Advanced Ceramics. Highlights the role of quantum materials (QMs) in artificial intelligence and the specific need for artificial synapses to match biological signal response.
  • Frañó, A., et al. (2023). "Non-locality in Quantum Materials." Nano Letters (referenced via UC San Diego News). This primary research breakthrough explains how electrical stimuli can affect non-neighboring electrodes, mimicking biological brain function.
  • Goh, K. E. J., et al. (2022). "Quantum Technologies for Engineering: the materials challenge." ResearchGate. Outlines the engineering-specific hurdles in scaling quantum hardware for practical use.
  • Marr, B. (2025). "7 Quantum Computing Trends That Will Shape Every Industry In 2026." Forbes. Provides context on the race for room-temperature quantum operations and the infrastructure challenges involved.
  • Schuller, I. K., et al. (2022). "Neuromorphic computing: Challenges from quantum materials to systems." Applied Physics Letters. Discusses the "holistic rethinking" required for computation and the energy-efficiency limitations of conventional semiconductors.
  • Zhang, H., et al. (2018). "Quantum materials pave the path for synthetic neuroscience." MRS Bulletin. Details how vanadium oxide and samarium nickelate act as tunable materials to replicate neuronal firing and synaptic gatekeeping.

Saturday, November 8, 2025

Industry derived holistic career development process for academic institutions

This article deals with ideas on converting the momentum required for industry lead training initiative in academia. The technique involves a  structured approach and is performed in different, distinct steps. Here is an explanation of a two-step (training & placement) view. This is toward skilling needs with the potential of students’ career development from a fresher to an early career professional.
 
  • LEVERAGING  MOMENTUM, NOT ADDING WEIGHT - An Industry perspective
 A worker needs to load a heavy tire onto a truck. Instead of relying on sheer strength, he takes a smarter route—rhythmically bouncing the tire, building energy and momentum until it rises to the right height. It’s not just about force—it’s about finesse.
 
Learning works the same way. Instructional Systems Design isn’t about flooding learners with content. It’s about pacing, feedback, and interaction—each element intentionally sequenced to build momentum. Like the tire, each step helps lift the next. In learning and development, real impact comes from structure and flow, not from more content or greater pressure. Thoughtful design delivers knowledge with purpose, encourages engagement, and keeps learners moving forward.  The best learning experiences respect cognitive limits. They don’t create friction—they create lift.
 
The industry's internal development programs are structured according to the aforementioned logic of distributing responsibilities across roles. This, in turn, is customized business-wise to yield maximum benefit for scaling training and development, rather than burning out employees. A similar analogy to be applied in the academic setting is what is derived in the next section of this work.
 
  • LEVERAGING  POTENTIAL, NOT ACCESSING PERFORMANCE - An Academic Perspective
Training as a system thinking activity occurs daily. And specifically, if it happens during the adult learning stages, with a set framework, the outcomes are just the evolution itself. Thus, within the education institutions, the training program is designed according to the industry requirements with great collaboration with experts as part of the board of studies panel meeting twice a year.  An effort is also made to develop communication skills and group discussion skills so that learners can perform better in a team. In this regard, 60 hours of intensive in-house training for a year is implemented along with the academic training (AT)
 
Job readiness year on year shall be achieved with a well-rounded placement process. A pre-placement training (PT) shall be organized in the summer break for n-1 years of students' regular degree programs, just before the final nth year 1st semester. A highly structured, customized 30-hour training shall be imparted every year. Topics such as Quantitative Ability, Verbal Ability, Logical Reasoning, resume preparation mock interviews shall be covered as well.
 
Quality domain-specific interviewing techniques training sessions are to be conducted to gear up the students for group discussions on various specializations. This program enables the students to acquire sufficient knowledge to qualify in their own domains, if any, written tests of various industry requirements, and every sort of company that shall be visiting the campus during the final/ nth year’s semester onwards.
 
Guest lectures, industry visits, and special invited lecture series, workshops are organized for all the branches of students from industry experts in collaboration with respective schools through regular graduate program syllabus implementation. These trainings are aimed at improving the placement percentage year on year.
 The Training Module shall consist of the following

            Domain-specific practice and tests
            Communication Skills
            General Aptitude practice and tests
            Quantitative Reasoning
            Logical Reasoning
            Verbal Reasoning
            Group Discussion
            Interview Skills

 Here is the high-level process of the training and placement process.

 
LEVERAGING  AVENUES, NOT ONE SIZE FITS ALL
 
During the lifecycle of a student in academics, it is bound responsibility of the institutions to be a partner for every student to set the course direction of a specialized career track for each and every individual. The opportunities are looked at from graduates pursuing post-graduation, or graduates seeking employment skills for jobs. Then there is also another category of graduates who have ideas in the vein of creating something for scale to get jobs out of their entrepreneurial spirit.

 The fundamental trait for the avenues starts with communication through languages. In the context of English reading, writing, and practice is in the main flow. As the medium has are varied nature, a common language eases interpersonal communication.

 The matching of patterns in day-to-day design and such logical thinking goes under aptitude thinking. This career development exercise is to train the brain constantly to practice the logic between nature and presence. Context-based domains dominate the industry, from finance, marketing, commerce, Information technology, automotives, and supply chains, to name a few. The skills related to each of these is the need for enterprise knowledge adoption in the industry.

 CONCLUSION

The integration of an industry-derived framework into academic career development is not merely an enhancement of existing programs; it represents a paradigm shift in preparing students for the modern workforce. By moving beyond traditional academic silos to incorporate practical skills, real-world problem-solving, and critical soft skills such as communication, teamwork, and adaptability, academic institutions can ensure their graduates are not just academically proficient but also immediately employable and career-ready. The research presented here establishes a strong correlation between holistic, industry-aligned development processes and enhanced student awareness of personalized career pathways and professional identity.

The key findings underscore the significance of training and development, such as soft skills and domain-based learning, in bridging the gap between academia and industry expectations. This approach fosters a symbiotic relationship, where institutions gain valuable market insights to inform curriculum development and employers benefit from a pipeline of talent equipped with demand-driven skills. Acknowledging the cultural and structural differences between academic and job environments is crucial, and successful implementation requires strong commitment from both educational leadership and industry partners.

While this study focused on the development and initial implementation of such a model, its broader implications suggest a future where the current model serves as a foundation for sustainable, lifelong learning. Future research should explore the long-term career trajectories of students who participated in these holistic programs, using multi-institutional cohorts to improve generalizability and refine the framework. Ultimately, by embracing this integrated approach, academic institutions can empower students to navigate their professional lives effectively, equipping them with the versatility and resilience needed for sustained success in an ever-evolving global market. The time to transition from an education-centric to a career-centric model, guided by industry needs, is now, ensuring that higher education remains a vital engine for both personal fulfilment and professional growth.
 



Labels: ,