AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks(2023)

objective:

• check the current challenges & limitations in AutoML field , think of relation between data-science pipeline
• what can LLMs do in the data-science pipeline

LLMs for AutoML

Opportunity_1. Improving Human-Machine Interaction with LLM

1. leverage LLMs as a user-friendly interface to AutoML
2. to improve the interpretability of the AutoML process

1.1. LLMs as an Interface to AutoML

limitations of og AutoML:

• were not built around user, but rather algorithmic ideas
• limited interaction with user in practice
• can not fully democratize ML process to common users (still need the experts)
• thus, many AutoML frameworks were objected to help datascientists(experts)
• 2 main target group of AutoML:
  • Domain Experts with low ML knowledge who want to apply ML to their problem
  • ML experts who want to improve their workflow with automated tools (much more targeted)

as LLM, rethink the concept of interaction between people and AutoML

• design powerful interactive text-based interfaces such as chatbot
• LLMs can simplify many complicated design of current AutoML process
  • choose an appropriate metric for optimization will be challenging for non-experts , but chatbot can guide them
  • similar concept with Github Copilot , but we envision much more ML-centered assistant

1.2. Enhance Interpretability of AutoML process

recent works tried to contribute to the human-centered AutoML paradigm(improve interaction & interpretability)

• “SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization”. In: Journal of Machine Learning Research 23.54, pp. 1–9 (2022).
• “Automated Benchmark-driven Design and Explanation of Hyperparameter Optimizers”. In: Evolutionary Computation,(2022)
• Explaining Hyperparameter Optimization via Partial Dependence Plots”. In: Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS’21).
• “Symbolic Explanations for Hyperparameter Optimization”. In: Proceedings of the Second International Conference on Automated Machine Learning(2023)

but many of these works just adapt classic interpretable ML methods to the AutoML setting

→ remain complicate to understand for non-experts and do not provide any textual , easy-to-understand explanations

why LLMs are powerful? It can Provide TEXTUAL EXPLANATION to human

⇒ LLM initialized with history of evaluated configurations or pipelines, a run history from an optimizer such as SMAC, Hyperopt, Botorch , context can help generate a textual optimization report elaborating with the final AutoML result and details of the process itself. Ideally, LLM can be contextualized with results of sevral AutoML methods as a strong foundation for the report.

user can also ask about the additional info to LLM AGENT which is not indicated in the final report.

 

 

Opportunity_2. LLMs for Configuring(setting) AutoML

example of AutoML process using LLM based on Bayesian Optimization

Configuring with LLM

remove the burden of selecting & configuring an AutoML system is needed, still a challenge with out experts. *configuring: empirical하게 search해야하는 대부분의 train setting들

recent works:

• “Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning”. In: Journal of Machine Learning Research(2022)
• “Towards Further Automation in AutoML”. In: ICML workshop on Automated Machine Learning (AutoML workshop 2018).
• “Automated Benchmark-driven Design and Explanation of Hyperparameter Optimizers”. In: Evolutionary Computation(2022)
• “Algorithm selection on a meta level”. In: Machine Learning(2022)

AutoML tools require the configuration of a search space of candidates from which solutions can be drawn. like hyperparam settings, which algorithm to choose, whether preprocessing or not, , .. etc

ex) whether to search optimal hyperparams in small domain(작은범위에서의 서치) or not sholud be decided by considering the trade-off of search speed & true optimal value

← Until now, either solved by experts( hard and time-consuming even for experts )

or by integrating knowledge of human to guide the search , recent works:

• “πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization”. In: Proceedings of the International Conference on Learning Representations (ICLR’22).
• “PriorBand: HyperBand + Human Expert Knowledge”. In: NeurIPS 2022 Workshop on Meta-Learning
• “Bayesian Optimization with a Prior for the Optimum”. In: Machine Learning and Knowledge Discovery in Databases (ECML/PKDD’21)

⇒ LLM can remove this kinds of burden by suggesting good configuration for the task at hand.

How long to run the AutoML to find optimal setting is also important in the perspective of Green AutoML. (long run time → high resource)

⇒ LLM can provide initial setting for maximun runtimes of AutoML tool based on the experience of other AutoML practitioners. / if go for the further use of LLM, can even leverage the knowledge of LLM to adjust the maximun runtime during the run by analyzing whether any additional optimization might still make sense or there’s no room for better solution.

 

LLM used for configuring the use of multi-fidelity approach such as Hyperband.

*Multi-Fidelity approach : refers to a method of utilizing models that operate at different levels of budget or resources during the model training and exploration process. ex) running models at different budget levels to find the optimal solution.

recent work about the multi-fidelity → performance influenced by the choice of the fidelity types and the min/max amount of budget.

 

 

Opportunity_3. LLM as Components of AutoML systems

 

LLM as sub-system

AutoML → plethora of sub-systems and components that suits differ purposes.

estimating the performance of a pipeline

• “How Powerful are Performance Predictors in Neural Architecture Search?” In: Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS’21).

estimating the runtime of a pipeline

• “Predicting Machine Learning Pipeline Runtimes in the Context of Automated Machine Learning”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence

⇒ LLM as meta-learned version of different sub-systems, which are used in for example, estimating the performance of a pipeline, estimating the runtime of a pipeline, or choosing the next pipeline to evaluate.

3.1. candidate selection

typical AutoML use candidate selection strategy (selects the next candidate to be evaluated, such as BO-based systems)

⇒ LLM can replace these mostly hand-designed selection strategies with a meta-learned version

GPT-NAS leverages a GPT model to predict (parts) of a neural architecture,

• “Gpt-nas: Neural Architecture Search With the Generative Pre-trained Model”. In: arXiv:2305.05351(2023)

GENIUS even goes a step further and replaces the whole architecture suggestion step with GPT-4.

• “Can Gpt-4 Perform Neural Architecture Search?” In: arXiv:2304.10970(2023).

EvoPrompting leverages an LLM to implement the crossover and mutation operator in an evolutionary NAS approach.

• “EvoPrompting: Language Models for Code-level Neural Architecture Search”. In: arXiv:2302.14838(2023)

3.2. feature engineering

LLM used for auto feature engineering (CAAFE) , Automating Data Science: Prospects and Challenges(2022, ACM)

3.3. evalutation process

typical AutoML evaluate new solution candidates by training and validating, and its time-consuming. (especially like DL, which has much longer train time )

→ some systems leverage meta-learned perfomance estimators such as meta-learned surrogate models in Bayeisan optimization(“Meta-Learning”. In: Automated Machine Learning: Methods, Systems, Challenges(2019)),

or neural performance predictors in NAS (“How Powerful are Performance Predictors in Neural Architecture Search?” In: Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS’21).)

⇒ LLM can access to much larger knowledge base and can do the better meta-learned replacement of traditional performance estimators. even can serve as a generator for training data for simpler performace/training time estimators or surrogate models.

3.4. zero-shot tools, only use LLMs to simulate the entire AutoML process

Given a textual problem description by the user and a knowledge base in the background, they suggest a pipeline and/or training procedure to achieve good performance.

(Going even further, some suggest AutoML-GPT and MLCopilot, respectively, which fully work as an zero-shot AutoML tool on their own)

• “MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks”. In: arXiv:2304.12210(2023)
• “AutoML-GPT: Automatic Machine Learning With GPT”. In: arXiv:2305.02499(2023)