Entropy optimality: Chang’s Lemma

In the last post, we saw how minimizing the relative entropy allowed us to obtain a “simple” feasible point of a particular polytope. We continue exploring this theme by giving a related proof of Chang’s Lemma, a Fourier-analytic result that arose in additive combinatorics. Our proof will reveal another aspect of entropy optimality: “Projected sub-gradient descent” and a corresponding convergence analysis.

1.1. Discrete Fourier analysis and Chang’s Lemma

Fix ${n \geq 1}$ and a prime ${p}$ . Let ${\mathbb F_p}$ denote the finite field with ${p}$ elements. We will work over ${\mathbb F_p^n}$ . For convenience, denote ${G=\mathbb F_p^n}$ . Let ${\mu}$ denote the uniform measure on ${G}$ , and for a non-negative function ${f : G \rightarrow [0,\infty)}$ , define the relative entropy

$\displaystyle \mathrm{Ent}_{\mu}(f) = \mathbb E_{\mu} \left[f \log \frac{f}{\mathbb E_{\mu} f}\right]\,,$

where we use the shorthand ${\mathop{\mathbb E}_{\mu} [f] = \sum_{x \in G} \mu(x) f(x)}$ . If ${\mathop{\mathbb E}_{\mu} [f] = 1}$ , we say that ${f}$ is a density (with respect to ${\mu}$ ). In that case, ${\mathrm{Ent}_{\mu}(f) = D(f \mu \,\|\,\mu)}$ where ${D}$ denotes the relative entropy (introduced in the last post).

We briefly review some notions from discrete Fourier analysis. One could skip this without much loss; our proof will only use boundedness of the Fourier characters and no other features. For functions ${f,g : G \rightarrow \mathbb C}$ , we will use the inner product ${\langle f,g\rangle = \mathop{\mathbb E}_{\mu}[f\bar g]}$ and the norms ${\|f\|_p = (\mathop{\mathbb E}_{\mu}[|f|^p])^{1/p}}$ for ${p \geq 1}$ .

For an element ${\alpha \in G}$ , we use

$\displaystyle \chi_{\alpha}(x) = \exp\left(\frac{-2\pi}{p} \sum_{i=1}^n \alpha_i x_i\right)$

to denote the corresponding Fourier character. Now every ${f : G \rightarrow \mathbb C}$ can be written in the Fourier basis as

$\displaystyle f = \sum_{\alpha \in G} \hat f(\alpha) \chi_{\alpha}\,,$

where ${\hat f(\alpha) = \langle f,\chi_{\alpha}\rangle}$ . (For simplicity of notation, we have implicitly identified ${G}$ and ${\hat G}$ .)

We will be interested in the large Fourier coefficients of a function. To that end, for ${\eta > 0}$ , we define

$\displaystyle \Delta_{\eta}(f) = \left\{ \alpha \in G : |\hat f(\alpha)| > \eta \|f\|_1 \right\}\,,$

Now we are ready to state Chang’s Lemma; it asserts that if ${\mathrm{Ent}_{\mu}(f)}$ is small, then all of ${f}$ ‘s large Fourier coefficients are contained in a low-dimensional subspace.

Lemma 1 If ${f : G \rightarrow [0,\infty)}$ is a density, then for every ${\eta > 0}$ the set ${\Delta_{\eta}(f)}$ is contained in an ${\mathbb F_p}$ -subspace of dimension at most ${d}$ , where

$\displaystyle d \leq 2 \sqrt{2} \frac{\mathrm{Ent}_{\mu}(f)}{\eta^2}\,.$

In the special case ${p=2}$ , the proof will actually give constant ${2}$ instead of ${2 \sqrt{2}}$ .

1.2. A slightly stronger statement

In fact, we will prove the following slightly stronger statement.

Theorem 2 If ${f : G \rightarrow [0,\infty)}$ is a density, then for every ${\eta > 0}$ there exists a density ${g : G \rightarrow [0,\infty)}$ such that ${\Delta_{\eta}(f) \subseteq \Delta_0(g)}$ and moreover ${g(x) = \Psi(\chi_{\alpha_1}(x), \ldots, \chi_{\alpha_d}(x))}$ for some ${d \leq 1+2\sqrt{2} \frac{\mathrm{Ent}_{\mu}(f)}{\eta^2}}$ and ${\{\alpha_i\} \subseteq G}$ and some function $\Psi : \mathbb C^d \to \mathbb R$ . In other words, ${g}$ is a function of at most ${d}$ characters.

It is an exercise in linear algebra to see that for such a density ${g}$ , the set of non-zero Fourier coefficients ${\Delta_0(g)}$ is contained in the span of ${\alpha_1, \ldots, \alpha_d}$ . Since ${\Delta_{\eta}(f) \subseteq \Delta_0(g)}$ , this yields Lemma 1.

1.3. Entropy optimality

Let ${L^2(G)}$ denote the Hilbert space of real-valued functions on ${G}$ equipped with the inner product ${\langle \cdot, \cdot\rangle}$ . Assume now that ${f : G \rightarrow [0,\infty)}$ is a density and fix ${\eta > 0}$ .

Dtnote by ${\mathcal F}$ the family of functions of the form ${\pm \mathrm{Re}\,\chi_\alpha}$ or ${\pm \mathrm{Im}\, \chi_\alpha}$ for some ${\alpha \in G}$ .

Let us associate to ${f}$ the convex polytope ${P(f, \eta) \subseteq L^2(G)}$ of functions ${g : G \rightarrow [0,\infty)}$ satisfying

$\displaystyle \langle g,\varphi\rangle \geq \langle f,\varphi\rangle - \eta$

for all ${\varphi \in \mathcal F}$ .

Just as in the matrix scaling setting, ${P(f,\eta)}$ encapsulates the family of “tests” that we care about (which Fourier coefficient magnitudes are large). Note that if ${g \in P(f,\eta/\sqrt{2})}$ , then ${\Delta_{\eta}(f) \subseteq \Delta_0(g)}$ . We also let ${Q}$ denote the probability (density) simplex: The set of all ${g : G \rightarrow [0,\infty)}$ such that ${\mathop{\mathbb E}_{\mu} g = 1}$ .

Of course, ${P(f,\eta) \cap Q}$ is non-empty because ${f \in P(f,\eta) \cap Q}$ . As before, we will try to find a “simple” representative from ${P(f,\eta) \cap Q}$ by doing entropy maximization (which is the same thing as relative entropy minimization):

$\displaystyle \textrm{minimize } \mathrm{Ent}_{\mu}(g) = \sum_{x \in G} \mu(x) g(x) \log g(x) \qquad \textrm{ subject to } g \in P(f,\eta) \cap Q\,.$

1.4. Sub-gradient descent

Soon we will explore this optimization using convex duality. But in this post, let us give an algorithm that tries to locate a feasible point of ${P(f,\eta) \cap Q}$ by a rather naive form of (sub-)gradient descent.

We start with the initial point ${g_0 = \mathbf{1}}$ which is the function that is identically ${1}$ on all of ${G}$ . Note that by strict convexity, this is the unique maximizer of ${\mathrm{Ent}_{\mu}(g)}$ among all ${g \in Q}$ . But clearly it may be that ${g_0 \notin P(f,\eta)}$ .

So suppose we have a function ${g_t \in Q}$ for some ${t \geq 0}$ . If ${g_t \in P(f,\eta)}$ , we are done. Otherwise, there exists a violated constraint: Some ${\varphi_t \in \mathcal F}$ such that ${\langle g_t, \varphi_t\rangle < \langle f,\varphi_t\rangle - \eta}$ . Our goal is now to update ${g_t}$ to ${g_{t+1} \in Q}$ that doesn't violate this constraint quite as much.

Notice the dichotomy being drawn here between the “soft'' constraints of ${P(f,\eta)}$ (which we try to satisfy slowly) and the “hard'' constraints of ${Q}$ that we are enforcing at every step. (This procedure and the analysis that follows are a special case of the mirror descent framework; see, for example, Section 4 of Bubeck’s monograph. In particular, there is a more principled way to design the algorithm that follows.)

We might try to write ${g_{t+1} = g_t + \varepsilon \varphi_t}$ for some small ${\varepsilon > 0}$ , but then ${g_{t+1}}$ could take negative values! Instead, we will follow the approximation ${e^x \approx 1+x}$ (for ${x}$ small) and make the update

$\displaystyle g_{t+1} = \frac{g_t \exp\left(\varepsilon \varphi_t\right)}{\mathbb E_{\mu}[g_t \exp\left(\varepsilon \varphi_t\right)]}\,.$

Observe that the exponential weight update keeps ${g_{t+1}}$ positive, and our explicit renormalization ensures that ${\mathop{\mathbb E}_{\mu} [g_{t+1}]=1}$ . Thus ${g_{t+1} \in Q}$ .

Of course, one might feel fishy about this whole procedure. Sure, we have improved the constraint corresponding to ${\varphi_t}$ , but maybe we messed up some earlier constraints that we had satisfied. While this may happen, it cannot go on for too long. In the next post, we will prove the following.

Lemma 3 For an appropriate choice of step size ${\varepsilon > 0}$ , it holds that ${g_T \in P(f,\eta)}$ for some ${T \leq 2 \frac{\mathrm{Ent}_{\mu}(f)}{\eta^2}}$ .

Notice that this lemma proves Theorem 2 and thus finishes our proof of Chang’s Lemma. That’s because by virtue of ${g_T \in P(f,\eta/\sqrt{2})}$ , we know that ${\Delta_{\eta}(f) \subseteq \Delta_0(g_T)}$ . Moreover, ${g_T}$ is built out of at most ${T+1}$ characters ${\alpha_0, \alpha_1, \ldots, \alpha_{T}}$ , and thus it satisfies the conclusion of Theorem 2.

For the case ${p=2}$ , we do not have to approximate the real and complex parts separately, so one saves a factor of ${\sqrt{2}}$ .

1.5. Bibliographic notes

This argument bears some similarity to the proof of Impagliazzo, Moore, and Russell. It was discovered in a joint work with Chan, Raghavendra, and Steurer on extended formulations. In both those arguments, the “descent” is implicit and is done by hand—the characters are carefully chosen not to interact. It turns out that this level of care is unnecessary. The analysis in the next post will allow the descent to choose any violated constraint at any time, and moreover will only use the fact that the characters are bounded.

tcs math

some mathematics & computation

Entropy optimality: Chang’s Lemma

Like this:

Related

Leave a Reply Cancel reply