Properly Learning Decision Trees in almost Polynomial Time

Blanc, Guy; Lange, Jane; Qiao, Mingda; Tan, Li-Yang

doi:10.1145/3561047

Properly Learning Decision Trees in almost Polynomial Time

Journal Article · Thu Nov 24 04:00:00 EST 2022 · Journal of the Association for Computing Machinery

DOI:https://doi.org/10.1145/3561047· OSTI ID:2421027

^[1]; ^[2]; ^[3]; ^[3]

Stanford, Stanford, CA, USA; OSTI
MIT, Cambridge, MA, USA
Stanford, Stanford, CA, USA

We give ann^{O(log logn)}-time membership query algorithm for properly and agnostically learning decision trees under the uniform distribution over { ± 1}ⁿ. Even in the realizable setting, the previous fastest runtime wasn^O(logn), a consequence of a classic algorithm of Ehrenfeucht and Haussler.

Our algorithm shares similarities with practical heuristics for learning decision trees, which we augment with additional ideas to circumvent known lower bounds against these heuristics. To analyze our algorithm, we prove a new structural result for decision trees that strengthens a theorem of O’Donnell, Saks, Schramm, and Servedio. While the OSSS theorem says that every decision tree has an influential variable, we show how every decision tree can be “pruned” so thateveryvariable in the resulting tree is influential.

Research Organization:: Stanford Univ., CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

DOE Contract Number:: SC0019205

OSTI ID:: 2421027

Journal Information:: Journal of the Association for Computing Machinery, Journal Name: Journal of the Association for Computing Machinery Journal Issue: 6 Vol. 69; ISSN 0004-5411

Publisher:: Association for Computing Machinery (ACM)

Country of Publication:: United States

Language:: English

References (19)

Learning functions of k relevant variables Mossel, Elchanan; O'Donnell, Ryan; Servedio, Rocco A. Journal of Computer and System Sciences, Vol. 69, Issue 3 https://doi.org/10.1016/j.jcss.2004.04.002	journal	November 2004
Learning Decision Trees Using the Fourier Spectrum Kushilevitz, Eyal; Mansour, Yishay SIAM Journal on Computing, Vol. 22, Issue 6 https://doi.org/10.1137/0222080	journal	December 1993
Learning and Smoothed Analysis Kalai, Adam Tauman; Samorodnitsky, Alex; Teng, Shang-Hua 2009 50th Annual IEEE Symposium on Foundations of Computer Science https://doi.org/10.1109/FOCS.2009.60	conference	October 2009
Constructing optimal binary decision trees is NP-complete Hyafil, Laurent; Rivest, Ronald L. Information Processing Letters, Vol. 5, Issue 1 https://doi.org/10.1016/0020-0190(76)90095-8	journal	May 1976
Finding Correlations in Subquadratic Time, with Applications to Learning Parities and the Closest Pair Problem Valiant, Gregory Journal of the ACM, Vol. 62, Issue 2 https://doi.org/10.1145/2728167	journal	May 2015
Approximating Optimal Binary Decision Trees Adler, Micah; Heeringa, Brent Algorithmica, Vol. 62, Issue 3-4 https://doi.org/10.1007/s00453-011-9510-9	journal	April 2011
Learning Monotone Decision Trees in Polynomial Time O'Donnell, Ryan; Servedio, Rocco A. SIAM Journal on Computing, Vol. 37, Issue 3 https://doi.org/10.1137/060669309	journal	January 2007
Finding Small Equivalent Decision Trees is hard Zantema, Hans; Bodlaender, Hans L. International Journal of Foundations of Computer Science, Vol. 11, Issue 02 https://doi.org/10.1142/S0129054100000193	journal	June 2000
Rank-r decision trees are a subclass of r-decision lists Blum, Avrim Information Processing Letters, Vol. 42, Issue 4 https://doi.org/10.1016/0020-0190(92)90237-P	journal	June 1992
Minimization of decision trees is hard to approximate Sieling, Detlef Journal of Computer and System Sciences, Vol. 74, Issue 3 https://doi.org/10.1016/j.jcss.2007.06.014	journal	May 2008
Beyond the low-degree algorithm: mixtures of subcubes and their applications Chen, Sitan; Moitra, Ankur Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing https://doi.org/10.1145/3313276.3316375	conference	June 2019
Selection of relevant features and examples in machine learning Blum, Avrim L.; Langley, Pat Artificial Intelligence, Vol. 97, Issue 1-2 https://doi.org/10.1016/s0004-3702(97)00063-5	journal	December 1997
Decision tree approximations of Boolean functions Mehta, Dinesh; Raghavan, Vijay Theoretical Computer Science, Vol. 270, Issue 1-2 https://doi.org/10.1016/S0304-3975(01)00011-1	journal	January 2002
Agnostically learning decision trees Gopalan, Parikshit; Kalai, Adam Tauman; Klivans, Adam R. Proceedings of the fortieth annual ACM symposium on Theory of computing https://doi.org/10.1145/1374376.1374451	conference	May 2008
Weakly learning DNF and characterizing statistical query learning using Fourier analysis Blum, Avrim; Furst, Merrick; Jackson, Jeffrey Proceedings of the twenty-sixth annual ACM symposium on Theory of computing - STOC '94 https://doi.org/10.1145/195058.195147	conference	January 1994
Sharp phase transition for the random-cluster and Potts models via decision trees Duminil-Copin, Hugo; Raoufi, Aran; Tassion, Vincent Annals of Mathematics, Vol. 189, Issue 1 https://doi.org/10.4007/annals.2019.189.1.2	journal	January 2019
Learning decision trees from random examples Ehrenfeucht, Andrzej; Haussler, David Information and Computation, Vol. 82, Issue 3 https://doi.org/10.1016/0890-5401(89)90001-1	journal	September 1989
Learning kμ decision trees on the uniform distribution Hancock, Thomas R. Proceedings of the sixth annual conference on Computational learning theory - COLT '93 https://doi.org/10.1145/168304.168374	conference	January 1993
Lower Bounds on Learning Decision Lists and Trees Hancock, Thomas; Jiang, Tao; Li, Ming Information and Computation, Vol. 126, Issue 2 https://doi.org/10.1006/inco.1996.0040	journal	May 1996

Similar Records

Inferring hierarchical clustering structures by deterministic annealing

Conference · Mon Dec 30 23:00:00 EST 1996 · OSTI ID:421314

Polynomial time approximation schemes for Euclidean TSP and other geometric problems

Conference · Mon Dec 30 23:00:00 EST 1996 · OSTI ID:457631

A polynomial time primal network simplex algorithm for minimum cost flows

Conference · Mon Dec 30 23:00:00 EST 1996 · OSTI ID:416832

Related Subjects

Computer Science

Properly Learning Decision Trees in almost Polynomial Time

Citation Formats

References (19)

Similar Records

Related Subjects