Zeroth-Order Machine Learning: Fundamental Principles and Emerging Applications in Foundation Models

Part 1: Introduction of ZO-ML

Preliminary Concepts and Mathematical Foundations Basic mathematical tools and formulations Why ZO over FO: Limitations of Traditional Gradient-Based Optimization Emerging challenges and drawbacks of relying solely on FO gradientbased methods Survey of Practical Applications and Use Cases Overview of applications that benefit from ZOML

Part 2: Foundations of ZO-ML

Algorithmic Landscape of ZO-ML A rundown of primary algorithms and methods in ZOML Convergence and Query Complexity Understanding the provable properties of ZOML Scaling ZO-ML: Practical Techniques and Implementations Tips and tricks for ZOML algorithms at scale Extending ZO-ML across Learning Paradigms How does ZOML adapt to various ML paradigms? Break

Part 3: Applications of ZO-ML

Prompt Learning in FMs Fine-tuning and Personalization in FMs via ZO-ML ZO-ML in the Context of AI Robustness, Efficiency, and Automation

Part 4: Demo Expo

Introducing the ZO-ML Toolbox A guided tour of our specialized toolbox for ZOML Benchmarking with ZO algorithms An introduction to ZO performance metrics and benchmark applications Practical Demos: Utilizing ZOT for Parameter-Efficient Fine-Tuning (PEFT), and Adversarial Defense Live demonstrations showcasing the utility of ZOML

Part 5: Conclusion and Q&A

Wrap-Up: Key Takeaways from the Tutorial Future Horizons: SP and ML Opportunities and Challenges Resources for Deeper Exploration A curated list of essential ZOML resources