A polynomial-time algorithm for learning noisy linear threshold functions

Conference · Tue Dec 31 04:00:00 EST 1996

OSTI ID:457664

Blum, A; Frieze, A; Kannan, R; Vempala, S ^[1]

---

1. Carnegie Mellon Univ., Pittsburgh, PA (United States)

In this paper we consider the problem of learning a linear threshold function (a half space in n dimensions, also called a {open_quotes}perceptron{close_quotes}). Methods for solving this problem generally fall into two categories. In the absence of noise, this problem can be formulated as a Linear Program and solved in polynomial time with the Ellipsoid Algorithm (or Interior Point methods). On the other hand, simple greedy algorithms such as the Perceptron Algorithm seem to work well in practice and can be made noise tolerant; but, their running time depends on a separation parameter (which quantifies the amount of {open_quotes}wiggle room{close_quotes} available) and can be exponential in the description length of the input. In this paper, we show how simple greedy methods can be used to find weak hypotheses (hypotheses that classify noticeably more than half of the examples) in polynomial time, without dependence on any separation parameter. This results in a polynomial-time algorithm for learning linear threshold functions in the PAC model in the presence of random classification noise. Our algorithm is based on a new method for removing outliers in data. Specifically, for any set S of points in R{sup n}, each given to b bits of precision, we show that one can remove only a small fraction of S so that in the remaining set T, for every vector {nu}, max{sub x{element_of}T}({nu} {center_dot} x){sup 2} {le} poly(n,b){vert_bar}T{vert_bar}{sup -1} {summation}{sub x{element_of}T}({nu} {center_dot} x){sup 2}. After removing these outliers, we are able to show that a modified version of the Perceptron Learning Algorithm works in polynomial time, even in the presence of random classification noise.

• OSTI ID: 457664
• Report Number(s): CONF-961004--; CNN: Grant CCR-9357793; Grant CCR9225008; Grant CCR9528973
• Country of Publication: United States
• Language: English

Related Subjects

99 GENERAL AND MISCELLANEOUS
ALGORITHMS
COMPUTERS
LEARNING
MATHEMATICAL LOGIC
POLYNOMIALS