CDQuant: Accurate Post-training Weight Quantization of Large Pre-trained Models using Greedy Coordinate Descent (2024)

Pranav Ajit Nair
Google DeepMind
pranavajitnair@google.com&Arun Sai Suggala
Google DeepMind
arunss@google.com

Abstract

Large language models (LLMs)^†^†Work done when the authors were at Google Research. have recently demonstrated remarkable performance across diverse language tasks. But their deployment is often constrained by their substantial computational and storage requirements. Quantization has emerged as a key technique for addressing this challenge, enabling the compression of large models with minimal impact on performance. The recent GPTQ algorithm, a post-training quantization (PTQ) method, has proven highly effective for compressing LLMs, sparking a wave of research that leverages GPTQ as a core component. Recognizing the pivotal role of GPTQ in the PTQ landscape, we introduce CDQuant, a simple and scalable alternative to GPTQ with improved performance. CDQuant uses coordinate descent to minimize the layer-wise reconstruction loss to achieve high-quality quantized weights. Our algorithm is easy to implement and scales efficiently to models with hundreds of billions of parameters. Through extensive evaluation on the PaLM2 model family, we demonstrate that CDQuant consistently outperforms GPTQ across diverse model sizes and quantization levels. In particular, for INT2 quantization of PaLM2-Otter, CDQuantachieves a $10\%$ reduction in perplexity compared to GPTQ.

1 Introduction

Large language models (LLMs) have shown remarkable ability to handle various language tasks(Touvron etal., 2023; OpenAI, 2023; Google, 2023), but their widespread adoption is hampered by their substantial computational and memory demands. To tackle this challenge, researchers have explored techniques like quantization, pruning, and distillation, with quantization being a particularly promising avenue for reducing model size and inference time without significantly sacrificing performance(Miao etal., 2023).Quantization techniques broadly fall into two categories: post-training quantization (PTQ) and quantization-aware training (QAT). QAT, while potentially yielding better results, poses a significant challenge for LLMs due to the immense resources required to train these large models. As a result, a growing body of research has focused on PTQ, which is generally less computationally intensive and can be applied to large, pre-trained models.

A seminal work in PTQ for LLMs is GPTQ(Frantar etal., 2022), which introduced a one-shot weight quantization approach that minimizes the following layer-wise output reconstruction loss: $\|X(W-\hat{W})\|_{F}^{2}$ ,where $X=[\mathbf{x}_{1},\mathbf{x}_{2}\dots\mathbf{x}_{n}]$ represents the layer inputs and $W,\widehat{W}\in\mathbb{R}^{d_{\text{in}}\times d_{\text{out}}}$ are the original and quantized weight matrices, respectively. This method has sparked a wave of research in PTQ. Subsequent works have built upon GPTQ, addressing its limitations and improving it’s performance.For instance, SpQR(Dettmers etal., 2023) and OWQ(Lee etal., 2023) improve GPTQ by explicitly addressing outlier weights, maintaining them in full precision, while quantizing the remaining weights using GPTQ. This hybrid approach improves quantization accuracy, particularly at lower bit-widths. Another line of research focuses on transforming the weight space before applying GPTQ. For instance, QuIP(Chee etal., 2024), FrameQuant(Adepu etal., 2024) transform the weights into a more quantization-friendly space and apply GPTQ in the transformed space. AWQ(Lin etal., 2023), SmoothQuant(Xiao etal., 2023) reduce the effect of activation outliers by performing feature scaling before quantizing the weights using standard techniques.

Given the central role GPTQ plays in the landscape of PTQ methods for LLMs, improving its quality directly translates to better quantization across numerous techniques that build upon it.Consequently, in this work, we revisit the core optimization problem addressed by GPTQ and introduce a novel coordinate descent based algorithm, CDQuant, to minimize the objective. CDQuantis an iterative optimization technique that greedily selects coordinates to descend along in each iteration. This approach contrasts with GPTQ, which cycles through the coordinates only once, in a predetermined order, often leading to suboptimal solutions for the layer-wise objective (see Section2 for more details). CDQuantis simple to implement and scales effectively to models with billions of parameters.We further extend CDQuant to group/sub-channel quantization, a technique gaining popularity of late(Dettmers etal., 2021; Dettmers and Zettlemoyer, 2023). Extensive experiments on PaLM 2(Anil etal., 2023) demonstrate that CDQuantconsistently outperforms GPTQ across various model sizes and quantization precision levels. For example, for INT2 quantization of PaLM2-Otter, CDQuantachieves a $10\%$ reduction in perplexity compared to GPTQ.

2 Related Work

The literature on quantization is huge. In this section, we only review the works that are related to quantization of large pre-trained models, and those that are related to our work.

Weight-only Quantization.

Optimal Brain Quantization (OBQ)(Frantar and Alistarh, 2022), a post-training quantization (PTQ) technique, draws inspiration from the Optimal Brain Surgeon (OBS) framework proposed byLeCun etal. (1989). Frantar and Alistarh (2022) introduced exact yet efficient algorithms for implementing the OBS greedy strategy. However, OBQ remains computationally expensive and struggles to scale effectively to large language models (LLMs). Frantar etal. (2022) subsequently introduced GPTQ, which employs heuristics to accelerate the OBQ algorithm. Specifically, GPTQ updates coordinates once in a cyclic manner rather than the greedy approach used in OBQ. This modification significantly speeds up the algorithm but comes at the cost of reduced performance. In our work, we address this performance drop by proposing greedy coordinate descent algorithms that are both straightforward and easy to implement.

Recent works have sought to improve upon GPTQ. One popular strategy here is to identify and isolate problematic weights before applying GPTQ. For example, SpQR(Dettmers etal., 2023) isolates outlier weights and keeps them at full precision, quantizing only the remaining values using GPTQ. Similarly, OWQ(Lee etal., 2023) identifies weights that are highly sensitive to small perturbations and excludes them from the GPTQ quantization process.Another promising direction involves transforming the weights into a different basis before quantization. Both QuIP(Chee etal., 2024), and FrameQuant(Adepu etal., 2024) take this route and perform quantization in the transformed space using GPTQ. In all these approaches, we could potentially substitute GPTQ with CDQuantand expect to see further improvements in performance. Other promising methods such as AWQ(Lin etal., 2023), AffineQuant(Ma etal., 2024), suppress the effect of outliers in input activations by adjusting their scale and transferring it to weights. They then perform the standard MinMax quantization on the rescaled weights. One could replace MinMax with GPTQ, CDQuantto improve the performance of these techniques (see Table3 for a comparison of CDQuantwith AWQ).

While working on our paper, we came across QuantEase(Behdin etal., 2023), a parallel research effort sharing similar goals as ours for improving GPTQ. While both methods leverage the concept of coordinate descent, QuantEase adopts a cyclic approach for weight updates, whereas our work employs a greedy strategy. Although our experiments (AppendixF) indicate comparable performance between the two methods, our work extends beyond QuantEase by introducing specialized algorithms for group/sub-channel quantization, not just full-channel quantization. Additionally, we develop novel block coordinate descent algorithms that further improve the performance. That being said, both these works (QuantEase and ours) collectively highlight the potential of coordinate descent algorithms in outperforming GPTQ.

Weight+Activation Quantization.

SmoothQuant(Xiao etal., 2023), OS+(Wei etal., 2023, 2022) have a similar flavour as AWQ, but quantize both weights and activations after scaling them appropriately. OmniQuant(Shao etal., 2023) performs quantization of the entire transformer block in a single shot. This encompasses both activation and weight quantization. Furthermore, it subsumes SmoothQuant, OS+ by using both feature scaling and outlier suppression. QLLM(Liu etal., 2023) tackles the issue of outliers in the activations by splitting the outlier features into multiple sub-channels and then recombining them, effectively reducing their influence. QLLM also incorporates a fine-tuning step at the end, introducing low-rank weights into each layer of the LLM. LLM.int8()(Dettmers etal., 2022) quantizes both acativations and weights to $8$ -bits and also identifies outliers and stores them in full precision.

Efficient End-to-End Quantization.

To recover the drop in performance from quantizationChai etal. (2023); Dettmers etal. (2024); Hu etal. (2021) perform low-rank parameter-efficient fine-tuning.

3 CDQuant

Notation.

Throughout the paper, we denote vectors by bold faced letters ( $\mathbf{a}$ ), and matrices by capital letters ( $A$ ). $\|\mathbf{a}\|_{2}=\sqrt{\sum_{i}\mathbf{a}_{i}^{2}}$ is the Euclidean norm and $\|A\|_{F}=\sqrt{\sum_{i,j}A_{ij}^{2}}$ is the Frobenius norm of a matrix. $\mbox{diag}(\mathbf{a})$ represents a diagonal matrix with $\mathbf{a}$ as its diagonal entries. $d_{\text{in}},d_{\text{out}}$ denote the input, output dimensions of a layer. $W\in\mathbb{R}^{d_{\text{in}}\times d_{\text{out}}}$ is the weight matrix of the layer, and $X=[\mathbf{x}_{1},\mathbf{x}_{2}\dots\mathbf{x}_{n}]\in\mathbb{R}^{n\times d_{%\text{in}}}$ is the matrix containing $n$ datapoints that are sent as input to the layer. $H=X^{T}X$ is the Hessian matrix for our objective in Equation(1). $c$ denotes the number of bits of precision used in quantization. In quantization, we aim to represent $W$ as $Q\times\mbox{diag}(\mathbf{a})+\mathbf{1}\mathbf{b}^{T}$ , where $\mathbf{a},\mathbf{b}\in\mathbb{R}^{d_{\text{out}}}$ represent the scale and bias parameters and $Q\in\{0,1,\dots 2^{c}-1\}^{d_{\text{in}}\times d_{\text{out}}}$ is the quantized matrix.

Many existing PTQ techniques, including GPTQ, aim to solve the following layer-wise optimization objective: $\min_{\mathbf{a},\mathbf{b},Q}\|X(W-Q\times\mbox{diag}(\mathbf{a})-\mathbf{1}%\mathbf{b}^{T})\|_{F}^{2}$ .Observe that this problem breaks down into $d_{\text{out}}$ independent problems across the output dimension. So, in the sequel, we focus on the following problem of quantizing a $d_{\text{in}}$ -dimensional vector

\displaystyle\min_{a,b,\mathbf{q}}\|X(\mathbf{w}-a\mathbf{q}-b)\|_{2}^{2}.

(1)

For a fixed $(a,b)$ , this problem is called Integer Linear Regression problem. It turns out, finding optimal solutions to this problem is NP-hard(Chrétien and Corset, 2009; Park and Boyd, 2018). So, several works have designed heuristics to solve this problem(Nagel etal., 2020; Li etal., 2021; Frantar and Alistarh, 2022; Frantar etal., 2022; Hubara etal., 2021). Within the context of LLMs, GPTQ is perhaps the most popular among these techniques, as it scales efficiently to models with billions of parameters(Frantar etal., 2022). In this work, we aim to improve upon GPTQ by designing better heuristics, while maintaining its scalability aspect. To this end, we rely on performing coordinate descent on objective(1), which we describe below.

3.1 Greedy Coordinate Descent

1:Input: $T$ - coordinate descent steps, $X$ - input data matrix, $\mathbf{w}$ - vector to be quantized, $a$ - scale, $b$ - bias, $\mathbf{q}_{0}$ - initial estimate

2:Compute Hessian $H$ as: $H\leftarrow X^{T}X$

3:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow 2H(\mathbf{q}_{0}-a^{-1}(\mathbf{w}-b))$

4:for $t\in[1:T]$ do

5:Find the coordinate that leads to the largest reduction in loss

i^{*},r^{*}=\operatorname*{arg\,min}_{i\in\{0,1,\dots d_{\text{in}}-1\},r\in\{%0,1,\dots 2^{c}-1\}}(r-\mathbf{q}_{t-1,i})^{2}H_{i,i}+(r-\mathbf{q}_{t-1,i})%\mathbf{g}_{i}

6:Update gradient $\mathbf{g}$ as

\mathbf{g}\leftarrow\mathbf{g}+2(r^{*}-\mathbf{q}_{t-1,i^{*}})H_{i^{*},\cdot},

where $H_{i^{*},\cdot}$ is the $i^{*}$ column of $H$

7:Update $\mathbf{q}_{t-1}$ as

\mathbf{q}_{t}\leftarrow\mathbf{q}_{t-1}+(r^{*}-\mathbf{q}_{t-1,i^{*}})\mathbf%{e}_{i^{*}},

where $\mathbf{e}_{i^{*}}$ is the standard basis vector with $1$ in $i^{*}$ position and $0$ everywhere else

8:endfor

In this section, we assume we have suitable values for scale ( $a$ ) and bias ( $b$ ) parameters already at hand, and focus on optimizing $\mathbf{q}$ . For a more in-depth discussion of how we determine these values, please refer to the final part of the section. As the name suggests, in greedy coordinate descent, at each round, we find the coordinate that leads to the biggest reduction in the objective and descend along that coordinate. Letting ${\mathcal{L}}(\mathbf{q})\coloneqq\|X(\mathbf{w}-a\mathbf{q}-b)\|_{2}^{2}$ be the objective in Equation(1), we try to find a coordinate $i$ and value $r$ , such that updating the $i^{th}$ coordinate to $r$ gives the biggest reduction in loss

\min_{i,r}{\mathcal{L}}(\mathbf{q}+(r-\mathbf{q}_{i})\mathbf{e}_{i})-{\mathcal%{L}}(\mathbf{q}),

where $\mathbf{e}_{i}$ is the standard basis vector with $1$ in $i$ position and $0$ everywhere else. Luckily for us, this can be implemented extremely efficiently as we have analytical expressions for the objective. In particular, one can easily show that

{\mathcal{L}}(\mathbf{q}+(r-\mathbf{q}_{i})\mathbf{e}_{i})-{\mathcal{L}}(%\mathbf{q})=(r-\mathbf{q}_{i})^{2}H_{i,i}+(r-\mathbf{q}_{i})\mathbf{g}_{i},

where $H,\mathbf{g}$ are the Hessian and gradient of ${\mathcal{L}}$ evaluated at $\mathbf{q}$ .Algorithm1 describes this procedure. Observe that line 5 is the most expensive step of the algorithm. It requires finding the minimum value among $d_{\text{in}}\times 2^{c}$ possibilities, a task that is well-suited for parallelization on GPUs.

Extension to Block Coordinate Descent.

A natural extension to Algorithm1 is block coordinate descent (BCD), where multiple coordinates are updated simultaneously in each iteration. While a greedy approach to BCD could, in principle, optimize the objective much better, the computational cost becomes prohibitive. Specifically, updating $k$ coordinates at a time necessitates evaluating $(d_{\text{in}}\times 2^{c})^{k}$ possible combinations of coordinates and their corresponding values. To overcome this, we propose a randomized BCD strategy (Algorithm 2), where we randomly partition the coordinates into $d_{\text{in}}/k$ blocks, and only search over these blocks and their corresponding values. This significantly reduces the search space to a more manageable $d_{\text{in}}/k\times 2^{kc}$ possibilities, making the algorithm practical for larger models. In our experiments, we primarily set $k=2$ and work with $c\in\{2,3,4\}$ .

Initializing $a,b,\mathbf{q}$ .

To initialize $a,b,\mathbf{q}$ in Algorithms1,2, we introduce a technique called Optimal Weight Clipping (OWC), which draws inspiration from the Learnable Weight Clipping (LWC) mechanism used in OmniQuant(Shao etal., 2023). In OWC, we quantize weight $\mathbf{w}$ as follows

\displaystyle\mathbf{q}=\text{clamp}\left(\left\lfloor\frac{\mathbf{w}-b}{a}%\right\rceil,0,2^{c}-1\right),\quad a=\frac{\gamma(\max(\mathbf{w})-\min(%\mathbf{w}))}{2^{c}-1},\quad b=\min(\mathbf{w}).

(2)

Here, $\gamma\in[0,1]$ represents the clipping strength. We determine the optimal $\gamma$ by minimizing the following layer-wise loss:

\min_{\gamma\in[0,1]}\|X(\mathbf{w}-a\mathbf{q}-b)\|_{2}^{2}.

Note that while not explicitly stated, both $\mathbf{w},a$ in the above objective are implicitly dependent on $\gamma$ .This optimization can be efficiently solved using a simple grid search. In contrast, LWC(Shao etal., 2023) optimizes a different objective function, focusing on end-to-end quantization of entire transformer block using gradient-based techniques. It is worth noting that setting $\gamma=1$ in OWCrecovers the widely used MinMax quantization scheme, which is used in many existing quantization methods, including GPTQ(Frantar etal., 2022), SmoothQuant‘(Xiao etal., 2023). In our experiments, we observed that OWCprovides a much better initialization compared to MinMax quantization, for both GPTQ and CDQuant.

1:Input: $T$ - coordinate descent steps, $k$ - block size, $X$ - input data matrix, $\mathbf{w}$ - vector to be quantized, $a$ - scale, $b$ - bias, $\mathbf{q}_{0}$ - initial estimate

3.2 Extension to Sub-channel Quantization

In this section, we consider sub-channel (or group) quantization, which is a morefine-grained quantization that divides the weight vector $\mathbf{w}$ into multiple groups and assigns a quantization scale to each group(Dettmers etal., 2021; Dettmers and Zettlemoyer, 2023). Letting $g$ be the group size, we divide weight $\mathbf{w}$ into $d_{\text{in}}/g$ groups $\{\mathbf{w}^{(0)},\mathbf{w}^{(1)},\dots\mathbf{w}^{(d_{\text{in}}/g-1)}\}$ each of size $g$ , and quantize $\mathbf{w}^{(i)}$ as $a^{(i)}\mathbf{q}^{(i)}+b^{(i)}$ . To learn the optimal parameters for this sub-channel quantization, we solve the following optimization problem:

\displaystyle\min_{\{a^{(i)},b^{(i)},\mathbf{q}^{(i)}\}_{i=0}^{d_{\text{in}}/g%-1}}\big{|}\big{|}\sum_{i=0}X^{(i)}(\mathbf{w}^{(i)}-a^{(i)}\mathbf{q}^{(i)}-b%^{(i)})\big{|}\big{|}_{2}^{2}.

(3)

Here, $X^{(i)}$ represents the columns of $X$ corresponding to the indices within group $i$ . To solve this optimization problem, we employ a coordinate descent approach, similar to Algorithms1 and 2 described earlier. That is, given initial values for the scaling and bias parameters, we iteratively optimize the quantized representation $\mathbf{q}$ using coordinate descent. Due to space constraints, we present the resulting algorithms in AppendixC (see Algorithms4,5).

Initialization.

Next, we tackle the initialization of parameters $\{a^{(i)},b^{(i)},\mathbf{q}^{(i)}\}_{i=0}^{d_{\text{in}}/g-1}$ for our coordinate descent procedure. Our approach draws inspiration from the OWCalgorithm described above, adapting its core idea to this problem. In essence, we reframe the initialization problem as one of selecting optimal clipping strengths $\{\gamma^{(i)}\}_{i=0}^{d_{\text{in}}/g-1}$ for groups $\{0,\dots d_{\text{in}}/g-1\}$ . This leads us to the following problem

\displaystyle\min_{\gamma^{(0)},\dots\gamma^{(d_{\text{in}}/g-1)}}\big{|}\big{%|}\sum_{i=0}X^{(i)}(\mathbf{w}^{(i)}-a^{(i)}\mathbf{q}^{(i)}-b^{(i)})\big{|}%\big{|}_{2}^{2},

(4)

where

\mathbf{q}^{(i)}=\text{clamp}\left(\left\lfloor\frac{\mathbf{w}^{(i)}-b^{(i)}}%{a^{(i)}}\right\rceil,0,2^{c}-1\right),\ a^{(i)}=\frac{\gamma^{(i)}(\max(%\mathbf{w}^{(i)})-\min(\mathbf{w}^{(i)}))}{2^{c}-1},\ b^{(i)}=\min(\mathbf{w}^%{(i)}).

We use greedy coordinate descent to optimize Equation(4). In each iteration, we update the $\gamma^{(i)}$ that leads to biggest drop in loss. This procedure, which we call OWC-CD, is described in Algorithm3.

1:Input: $T$ - coordinate descent steps, $g$ - group size, $X$ - input data matrix, $\mathbf{w}$ - weight vector, $\Gamma$ - grid of possible values for clipping strength

2:for $\beta\in\Gamma$ do

3:for $i\in[0:d_{\text{in}}/g-1]$ do

4:Compute quantization residual $\Delta(i,\beta)$ for group $i$ with clipping strength $\beta$ as:

\Delta(i,\beta)\leftarrow\mathbf{w}^{(i)}-a^{(i)}(\beta)\mathbf{q}^{(i)}(\beta%)-b^{(i)}(\beta)

. where $a^{(i)}(\beta),b^{(i)}(\beta),\mathbf{q}^{(i)}(\beta)$ are as defined in Equation(2).

5:endfor

6:endfor

7:Initialize clipping strengths for each group $\gamma^{(0)},\dots\gamma^{(d_{\text{in}}/g-1)}$

8:Compute Hessian $H$ as: $H\leftarrow X^{T}X$

9: $\mathbf{v}_{i}\leftarrow-2H_{i}(a\mathbf{q}+b-w)$ where $H_{i}$ is the sub-matrix of $H$ corresponding to the columns of group $i$ .

10:for $t\in[1:T]$ do

11:Find the group that leads to the largest reduction in loss

	$\displaystyle i^{},\beta^{}=\operatorname*{arg\,min}_{\begin{subarray}{c}i%\in\{0,1,\dots d_{\text{in}}/g-1\},\\\beta\in\Gamma\end{subarray}}$	$\displaystyle(\Delta(i,\gamma^{(i)})-\Delta(i,\beta))^{T}H_{i,i}(\Delta(i,%\gamma^{(i)})-\Delta(i,\beta))$
		$\displaystyle+\mathbf{v}_{i}^{T}(\Delta(i,\gamma^{(i)})-\Delta(i,\beta))$

where $H_{i,i}$ is the block diagonal element of $H$ corresponding to group $i$ .

12:Update $\mathbf{v}\leftarrow\mathbf{v}+2(\Delta(i^{*},\gamma^{(i^{*})})-\Delta(i^{*},%\beta^{*}))^{T}H_{i^{*}}$

13:Update $\gamma^{(i^{*})}\leftarrow\beta^{*}$

14:endfor

4 Experiments

Quantization.

In our experiments, we focus on weight-only quantization. We present results for two scenarios: (1) quantizing the FFN layers and (2) quantizing both the FFN and attention weights. In the former setting, all attention layers are quantized to INT8 using MinMax quantization, and low-bit quantization schemes are only applied to the feed-forward layers. It’s worth pointing out that this setting is very relevant in practice because FFN layers account for majority of the latency, and quantizing attention weights yields negligible latency reduction at the expense of model quality(Samaga etal., 2024). We perform both per-channel and sub-channel quantization (with a group size of $128$ ). We mainly focus on INT2, INT3 and INT4 quantization.

Models.

We use PaLM2 models(Anil etal., 2023) in all our experiments. In particular, we use PaLM2-Gecko, PaLM2-Otter, PaLM2-Bison models. These models vary in size, with PaLM2-Gecko being the smallest, followed by PaLM2-Otter, and finally PaLM2-Bison.

Baselines.

Since our primary goal is to demonstrate that CDQuantgets improved performance over GPTQ, we have chosen it as the primary baseline in most of our experiments. To further demonstrate the our technique can be used as a plug-and-play replacement for GPTQ, we also include comparisons with AWQ(Lin etal., 2023). For all our experiments, we initialize GPTQ with OWC, and run GPTQ for $T=d_{\text{in}}$ steps. To ensure stability and generalization, GPTQ regularizes its Hessian matrix by adding a scaled identity matrix ( $\lambda I$ ). Tuning this $\lambda$ for every (model, layer) pair is infeasible. So, we determine a single optimal value using the PaLM2-Gecko model, and apply it universally.

CDQuant.

We evaluate both the coordinate descent variants described in Section3.1: CD (Algorithm1), BCD (Algorithm2). For per-channel quantization, we initialize CD with Optimal Weight Clipping (OWC), and for sub-channel quantization, we additionally include initialization with OWC-CD. BCD is always initialized with CD in our experiments. Unless otherwise stated, both CD and BCD are run for $T=d_{\text{in}}$ iterations, OWC-CDis run for $d_{\text{in}}/g$ iterations, where $g$ is the group size. Similar to GPTQ, we regularize the Hessian matrix used in CD, BCD by adding $\lambda I$ . We determine a reasonable value for $\lambda$ using the PaLM2-Gecko model, and use it in all our experiments.

Evaluation.

Following recent works(Frantar etal., 2022; Ma etal., 2024), we evaluate all algorithms using two key metrics: perplexity and downstream task performance. To ensure a robust evaluation, we calculate perplexity using a $100$ million token subset derived from the PaLM2 training mixture. We chose to do this since the evaluation sets for PTB(Marcus etal., 1994) and WikiText2(Merity etal., 2017) datasets, that are commonly used in the field, are orders of magnitude smaller (83K and 246K tokens in their respective test sets, and 512K tokens subsampled from the C4 validation set (Frantar etal., 2022)), and thus can have high variance, and may not accurately reflect the model’s true performance. We use TriviaQA(Joshi etal., 2017), SQuAD(Rajpurkar etal., 2018), NaturalQuestions(Kwiatkowski etal., 2019) and WebQuestions(Berant etal., 2013) to evaluate generation capabilities of the quantized models, and to evaluate their reasoning capabilities, we test on ARC-c,ARC-e(Clark etal., 2018), HellaSwag(Zellers etal., 2019), BoolQ(Clark etal., 2019), PIQA(Bisk etal., 2020) and WinoGrande(Sakaguchi etal., 2020). For downstream evaluations, we consider both zero-shot, and one-shot settings. Finally, to determine which optimization technique is most effective at solving the layer-wise objective in Equation(1), we evaluate the final solution’s objective value using the same $100$ million tokens that we used to compute perplexity.

Training.

All techniques are calibrated using $1280$ data points, where each data point has $2048$ tokens. For OWC, we use a grid size of $50$ to find the most optimal $\gamma$ . We used 8 Nvidia H $100$ GPUs for quantizing the models.

Config	Method	PaLM2-Gecko	PaLM2-Otter	PaLM2-Bison
w16a16	-	7.948	5.984	5.298
w3a16	OWC	$12.570$	$17.928$	$6.169$
w3a16	GPTQ	$11.347$	$7.176$	$5.774$
	CD	$10.920$	$7.002$	$5.739$
	BCD(k=2)	$\mathbf{10.898}$	$\mathbf{6.979}$	$\mathbf{5.733}$
w3a16g128	OWC	$11.597$	$8.342$	$5.847$
w3a16g128	GPTQ	$10.414$	$6.635$	$5.677$
	CD	$10.273$	$6.655$	$5.656$
	BCD(k=2)	$10.259$	$6.545$	$5.654$
	OWC-CD	$10.706$	$6.635$	$5.686$
	OWC-CD+ CD	$10.143$	$6.528$	$5.650$
	OWC-CD+ BCD(k=2)	$\mathbf{10.138}$	$\mathbf{6.527}$	$\mathbf{5.647}$
w4a16	OWC	$8.946$	$6.693$	$5.475$
w4a16	GPTQ	$8.764$	$6.249$	$5.417$
	CD	$8.694$	$6.195$	$5.407$
	BCD(k=2)	$\mathbf{8.691}$	$\mathbf{6.192}$	$\mathbf{5.405}$
w4a16g128	OWC	$8.613$	$6.264$	$5.401$
w4a16g128	GPTQ	$8.498$	$6.112$	$5.377$
	CD	$8.456$	$6.097$	$5.373$
	BCD(k=2)	$8.454$	$6.097$	$5.372$
	OWC-CD	$8.519$	$6.106$	$5.377$
	OWC-CD+ CD	$8.436$	$6.092$	$5.371$
	OWC-CD+ BCD(k=2)	$\mathbf{8.434}$	$\mathbf{6.091}$	$\mathbf{5.371}$

Config	Method	PaLM2-Gecko			PaLM2-Otter			PaLM2-Bison
		Gen.	Rank	Avg.	Gen.	Rank	Avg.	Gen.	Rank	Avg.
w16a16	-	$25.85$	$61.44$	$47.2$	$44.31$	$75.43$	$62.98$	$51.08$	$78.28$	$68.00$
w3a16	OWC	$18.95$	$55.74$	$41.02$	$21.02$	$65.78$	$47.87$	$47.52$	$77.76$	$65.66$
w3a16	GPTQ	$19.42$	$57.71$	$42.39$	$37.26$	$73.95$	$59.27$	$48.55$	$78.58$	$66.57$
	CD	$20.69$	$58.05$	$43.11$	$38.12$	$73.37$	$59.27$	$48.47$	$78.56$	$66.53$
	BCD(k=2)	$20.52$	$58.44$	$43.27$	$38.53$	$73.46$	$59.49$	$48.08$	$78.66$	$66.43$
w3a16g128	OWC	$22.83$	$57.02$	$43.35$	$33.18$	$70.81$	$55.76$	$48.08$	$78.34$	$66.24$
w3a16g128	GPTQ	$22.24$	$58.66$	$44.09$	$40.30$	$74.12$	$60.59$	$49.15$	$78.58$	$66.81$
	CD	$22.79$	$58.81$	$44.40$	$41.33$	$74.27$	$61.10$	$49.17$	$78.49$	$66.76$
	BCD(k=2)	$22.64$	$58.85$	$44.37$	$41.19$	$73.94$	$60.84$	$49.02$	$78.51$	$66.71$
	OWC-CD	$22.21$	$58.69$	$44.10$	$40.84$	$74.33$	$60.93$	$48.54$	$78.29$	$66.39$
	OWC-CD+ CD	$22.44$	$58.84$	$44.28$	$40.87$	$74.41$	$61.00$	$49.22$	$78.86$	$67.00$
	OWC-CD+ BCD(k=2)	$22.15$	$58.77$	$44.13$	$41.19$	$74.65$	$61.27$	$49.08$	$78.3$	$66.61$
w4a16	OWC	$23.27$	$60.22$	$45.44$	$41.70$	$74.21$	$61.21$	$50.05$	$78.72$	$67.25$
w4a16	GPTQ	$24.51$	$60.72$	$46.24$	$43.00$	$74.86$	$62.12$	$50.28$	$79.15$	$67.6$
	CD	$24.60$	$60.62$	$46.22$	$43.21$	$75.17$	$62.39$	$50.09$	$79.21$	$67.56$
	BCD(k=2)	$24.52$	$60.62$	$46.18$	$43.16$	$75.50$	$62.56$	$50.25$	$79.14$	$67.59$
w4a16g128	OWC	$24.63$	$61.27$	$46.61$	$42.31$	$74.76$	$61.78$	$50.49$	$79.38$	$67.82$
w4a16g128	GPTQ	$25.40$	$61.13$	$46.84$	$43.54$	$75.02$	$62.43$	$50.63$	$79.21$	$67.78$
	CD	$25.18$	$61.42$	$46.92$	$43.78$	$74.93$	$62.47$	$50.73$	$79.30$	$67.87$
	BCD(k=2)	$24.95$	$61.49$	$46.87$	$43.77$	$75.04$	$62.53$	$50.84$	$79.56$	$68.07$
	OWC-CD	$24.59$	$61.40$	$46.67$	$43.67$	$75.31$	$62.65$	$50.65$	$79.04$	$67.69$
	OWC-CD+ CD	$25.12$	$61.44$	$46.91$	$44.00$	$75.27$	$62.76$	$50.62$	$79.21$	$67.78$
	OWC-CD+ BCD(k=2)	$25.00$	$61.50$	$46.90$	$43.75$	$75.20$	$62.62$	$50.64$	$79.17$	$67.76$

Results.

Table1 presents the perplexity numbers for different quantization techniques applied to FFN layers. It can be seen that both our CD and BCD methods have a clear advantage over GPTQ, leading to lower perplexity scores for all models and quantization levels. The difference is more pronounced for lower bit quantization. This is further highlighted in Table5, which focuses specifically on INT2 quantization. Here, we see almost $10\%$ improvement in perplexity over GPTQ. We also present (1-shot) downstream evals in Tables2,5 (for 0-shot results, and a detailed breakdown of results for each dataset, please refer to AppendixD). In Table5 we see $2$ - $2.5\%$ relative improvement in downstream evals over GPTQ. Furthermore, Table2 demonstrates that our methods consistently match or exceed GPTQ’s performance across all the settings, with the most substantial improvements observed in low-bit quantization, and smaller models. Finally, we also observe that coordinate descent techniques are better at optimizing the layer-wise objective in Equation(1), than GPTQ. For instance, the average objective value (relative to all $0$ ’s solution) for the GPTQ solution for the $1^{\text{st}}$ feed-forward layer is $0.164$ , whereas for CD it is $0.158$ , and for BCD(k= $2$ ) it is $0.157$ .

Next, observe that Table5 also investigates the effect of multiple epochs of CD and BCD, where each epoch corresponds to $d_{\text{in}}$ iterations. For BCD with $k=2$ , additional epochs enhance performance. However, for BCD with $k=4$ , multiple epochs appear to lead to overfitting, indicating a need for stronger regularization.

FFN+Attention Quantization. In addition to FFN quantization, we also perform FFN + attention weight quantization. However, the inputs to the attention layer are often aligned in a handful of directions. Consequently, performing quantization using such a data leads to a huge drop in performance, as the algorithms would primarily focus on a few directions and ignore the rest. To mitigate this, we clip the largest eigenvalues of the Hessian to ensure a more balanced Hessian. This technique, reminiscent of the weight clipping in OmniQuant(Shao etal., 2023),improves the performance of both GPTQ and CDQuant¹¹1One could also rely on existing techniques such as AWQ and SmoothQuant to reduce the effect of outliers. But in our experiments we noticed that both AWQ and SmoothQuant performed poorly compared to the simple eigenvalue clipping technique.. For instance, for PaLM2-Gecko, the perplexity for w3a16-GPTQ improves from $34.872$ to $24.054$ with clipping, and for w4a16-GPTQ, it improves from $10.966$ to $10.005$ . Table4 presents the results from this experiment, where we run all the algorithms on the clipped Hessian. We once again notice that in almost all the settings, our coordinate descent algorithms outperform GPTQ.

Method	PaLM2-Otter
	w3a16		w3a16g128
	with AWQ	w/o AWQ	with AWQ	w/o AWQ
OWC	$22.661$	$17.928$	$6.813$	$8.342$
GPTQ	$7.381$	$7.176$	$6.519$	$6.635$
CD	$7.244$	$7.002$	$6.475$	$6.655$
BCD(k=2)	$\mathbf{7.211}$	$\mathbf{6.979}$	$\mathbf{6.474}$	$\mathbf{6.545}$

AWQ. A major strength of our algorithm is its versatility. It seamlessly replaces GPTQ in any quantization technique that relies on it. To illustrate this, we focus on the AWQ algorithm. We demonstrate that our algorithm, when layered on top of AWQ, surpasses the performance of GPTQ layered on top of AWQ. Table3 presents the result from this experiment. It can be seen that BCD always outperforms GPTQ, for both per-channel and group quantization²²2The performance of AWQ on PaLM2-Bison is much worse than not performing AWQ. Hence, in Table3, we only present results for PaLM2-Otter..

Runtime.

The runtimes of various techniques for quantizing PaLM2-Otter and PaLM2-Bison are presented in Table6. To be precise, the table presents time taken to quantize all the FFN, attention layers in the model, for per-channel quantization. For group quantization runtimes, please refer to AppendixE. It can be seen that CD and GPTQ have comparable runtimes. While BCD takes longer than CD, the majority of its time is spent quantizing the $2^{nd}$ FFN layer, where the quantization dimension is equivalent to the model’s hidden dimension. So, in practice, to speed up the entire process, one could rely on CD for quantizing FFN2 and use BCD for quantizing the rest of the weight matrices.

Config	Method	PaLM2-Gecko	PaLM2-Otter	PaLM2-Bison
w16a16	-	7.948	5.984	5.298
w3a16	OWC	$41.751$	$54.499$	$6.446$
w3a16	GPTQ	$24.054$	$\mathbf{11.384}$	$5.801$
	CD	$19.692$	$11.542$	$5.770$
	BCD(k=2)	$\mathbf{19.312}$	$11.392$	$\mathbf{5.762}$
w3a16g128	OWC	$20.301$	$9.785$	$5.973$
w3a16g128	GPTQ	$15.042$	$7.501$	$5.698$
	CD	$14.757$	$7.454$	$5.679$
	BCD(k=2)	$14.691$	$\mathbf{7.438}$	$5.673$
	OWC-CD	$16.268$	$7.617$	$5.746$
	OWC-CD+ CD	$14.285$	$7.466$	$5.675$
	OWC-CD+ BCD(k=2)	$\mathbf{14.242}$	$7.458$	$\mathbf{5.670}$
w4a16	OWC	$10.683$	$7.194$	$5.508$
w4a16	GPTQ	$10.005$	$6.645$	$5.425$
	CD	$9.991$	$6.627$	$5.415$
	BCD(k=2)	$\mathbf{9.965}$	$\mathbf{6.621}$	$\mathbf{5.413}$
w4a16g128	OWC	$9.402$	$6.415$	$5.420$
w4a16g128	GPTQ	$9.160$	$6.241$	$5.381$
	CD	$9.124$	$6.226$	$5.377$
	BCD(k=2)	$9.120$	$\mathbf{6.225}$	$5.376$
	OWC-CD	$9.241$	$6.261$	$5.387$
	OWC-CD+ CD	$9.094$	$6.244$	$5.375$
	OWC-CD+ BCD(k=2)	$\mathbf{9.093}$	$6.242$	$\mathbf{5.374}$

Method	Epochs	PaLM2-Otter				PaLM2-Bison
w2a16g128		Perplexity	Generation	Rank	Avg.	Perplexity	Generation	Rank	Avg.
GPTQ	-	$10.816$	$26.98$	$67.14$	$51.07$	$7.230$	$39.32$	$74.19$	$60.24$
CD	$1$	$9.917$	$29.87$	$66.76$	$52.00$	$7.123$	$38.73$	$73.56$	$59.63$
BCD(k=2)	$1$	$9.822$	$30.03$	$66.82$	$52.11$	$7.094$	$38.85$	$73.99$	$59.93$
BCD(k=2)	$2$	$9.760$	$30.09$	$\mathbf{67.45}$	$\mathbf{52.51}$	$\mathbf{7.088}$	$\mathbf{39.35}$	$74.10$	$60.20$
	$3$	$9.719$	$\mathbf{30.19}$	$67.09$	$52.33$	$7.094$	$39.30$	$\mathbf{74.24}$	$\mathbf{60.26}$
BCD(k=4)	$1$	$\mathbf{9.708}$	$30.11$	$67.09$	$52.30$	$7.099$	$38.98$	$74.20$	$60.11$
BCD(k=4)	$2$	$9.752$	$30.00$	$66.70$	$52.02$	$7.099$	$39.09$	$74.08$	$60.08$
	$3$	$9.749$	$30.07$	$66.93$	$52.19$	$7.099$	$39.05$	$74.10$	$60.08$

Config

Method

PaLM2-Otter

PaLM2-Bison

FFN runtime

(in minutes)

Attn. runtime

(in minutes)

FFN runtime

(in minutes)

Attn. runtime

(in minutes)

w3a16

GPTQ

0.90

0.72

6.42

0.63

2.94

0.49

16.13

0.43

BCD(k=2)

14.03

2.08

82.13

1.81

w4a16

GPTQ

0.90

0.64

6.60

0.56

3.59

0.59

21.19

0.52

BCD(k=2)

34.85

4.72

235.14

4.13

5 Conclusion and Future Work

In this work, we developed a coordinate descent framework (CDQuant) for quantization of LLMs. CDQuantis a simple and effective alternative to GPTQ, that consistently outperformed it on PaLM2 models. The simplicity of our algorithm makes it a seamless substitute for GPTQ in various algorithmic contexts where GPTQ currently functions as a sub-routine. Our future work aims to further improve the performance of CDQuant. In particular, we aim to speed up our BCD algorithm, and make it as fast as CD. Furthermore, we will focus on developing layer-wise loss functions that are more closely aligned with end-to-end loss, thereby reducing the performance gap between full-precision and quantized models. Finally, we will also explore integrating CDQuantwith QuIP and FrameQuant to evaluate potential performance gains for extreme low-bit quantization.

Acknowledgements

We are grateful to Kyuyeun Kim, Wonpyo Park, and Jaehong Kim for assisting in setting up the calibration pipelines, and Prateek Jain for helpful discussions, support, and feedback.

References

Touvron etal. [2023]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023.
OpenAI [2023]OpenAI.Gpt-4 technical report.Technical report, 2023.URL https://cdn.openai.com/papers/gpt-4.pdf.
Google [2023]GeminiTeam Google.Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023.
Miao etal. [2023]Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia.Towards efficient generative large language model serving: A survey from algorithms to systems.arXiv preprint arXiv:2312.15234, 2023.
Frantar etal. [2022]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.
Dettmers etal. [2023]Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078, 2023.
Lee etal. [2023]Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park.Owq: Lessons learned from activation outliers for weight quantization in large language models.arXiv preprint arXiv:2306.02272, 2023.
Chee etal. [2024]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and ChristopherM DeSa.Quip: 2-bit quantization of large language models with guarantees.Advances in Neural Information Processing Systems, 36, 2024.
Adepu etal. [2024]Harshavardhan Adepu, Zhanpeng Zeng, LiZhang, and Vikas Singh.Framequant: Flexible low-bit quantization for transformers.arXiv preprint arXiv:2403.06082, 2024.
Lin etal. [2023]JiLin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.Awq: Activation-aware weight quantization for llm compression and acceleration.arXiv preprint arXiv:2306.00978, 2023.
Xiao etal. [2023]Guangxuan Xiao, JiLin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.Smoothquant: Accurate and efficient post-training quantization for large language models.In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023.
Dettmers etal. [2021]Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer.8-bit optimizers via block-wise quantization.arXiv preprint arXiv:2110.02861, 2021.
Dettmers and Zettlemoyer [2023]Tim Dettmers and Luke Zettlemoyer.The case for 4-bit precision: k-bit inference scaling laws.In International Conference on Machine Learning, pages 7750–7774. PMLR, 2023.
Anil etal. [2023]Rohan Anil, AndrewM Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, etal.Palm 2 technical report.arXiv preprint arXiv:2305.10403, 2023.
Frantar and Alistarh [2022]Elias Frantar and Dan Alistarh.Optimal brain compression: A framework for accurate post-training quantization and pruning.Advances in Neural Information Processing Systems, 35:4475–4488, 2022.
LeCun etal. [1989]Yann LeCun, John Denker, and Sara Solla.Optimal brain damage.Advances in neural information processing systems, 2, 1989.
Ma etal. [2024]Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji.Affinequant: Affine transformation quantization for large language models.arXiv preprint arXiv:2403.12544, 2024.
Behdin etal. [2023]Kayhan Behdin, Ayan Acharya, Aman Gupta, Qingquan Song, Siyu Zhu, Sathiya Keerthi, and Rahul Mazumder.Quantease: Optimization-based quantization for language models.arXiv preprint arXiv:2309.01885, 2023.
Wei etal. [2023]Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu.Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.arXiv preprint arXiv:2304.09145, 2023.
Wei etal. [2022]Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, QiZhang, Fengwei Yu, and Xianglong Liu.Outlier suppression: Pushing the limit of low-bit transformer language models.Advances in Neural Information Processing Systems, 35:17402–17414, 2022.
Shao etal. [2023]Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, YuQiao, and Ping Luo.Omniquant: Omnidirectionally calibrated quantization for large language models.arXiv preprint arXiv:2308.13137, 2023.
Liu etal. [2023]Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023.
Dettmers etal. [2022]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.Advances in Neural Information Processing Systems, 35:30318–30332, 2022.
Chai etal. [2023]Yuji Chai, John Gkountouras, GlennG Ko, David Brooks, and Gu-Yeon Wei.Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation.arXiv preprint arXiv:2306.08162, 2023.
Dettmers etal. [2024]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024.
Hu etal. [2021]EdwardJ Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, LuWang, and Weizhu Chen.Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021.
Chrétien and Corset [2009]Stéphane Chrétien and Franck Corset.Using the eigenvalue relaxation for binary least-squares estimation problems.Signal processing, 89(11):2079–2091, 2009.
Park and Boyd [2018]Jaehyun Park and Stephen Boyd.A semidefinite programming method for integer convex quadratic minimization.Optimization Letters, 12:499–518, 2018.
Nagel etal. [2020]Markus Nagel, RanaAli Amjad, Mart VanBaalen, Christos Louizos, and Tijmen Blankevoort.Up or down? adaptive rounding for post-training quantization.In International Conference on Machine Learning, pages 7197–7206. PMLR, 2020.
Li etal. [2021]Yuhang Li, Ruihao Gong, XuTan, Yang Yang, Peng Hu, QiZhang, Fengwei Yu, Wei Wang, and Shi Gu.Brecq: Pushing the limit of post-training quantization by block reconstruction.arXiv preprint arXiv:2102.05426, 2021.
Hubara etal. [2021]Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.Accurate post training quantization with small calibration sets.In International Conference on Machine Learning, pages 4466–4475. PMLR, 2021.
Samaga etal. [2024]Yashas Samaga, Varun Yerram, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Prateek Jain, and Praneeth Netrapalli.Hire: High recall approximate top-k estimation for efficient llm inference.ArXiv, abs/2402.09360, 2024.URL https://api.semanticscholar.org/CorpusID:267657774.
Marcus etal. [1994]MitchellP. Marcus, Grace Kim, MaryAnn Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger.The penn treebank: Annotating predicate argument structure.In Human Language Technology, Proceedings of a Workshop held at Plainsboro, New Jerey, USA, March 8-11, 1994. Morgan Kaufmann, 1994.URL https://aclanthology.org/H94-1020/.
Merity etal. [2017]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.Pointer sentinel mixture models.In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.URL https://openreview.net/forum?id=Byj72udxe.
Joshi etal. [2017]Mandar Joshi, Eunsol Choi, DanielS. Weld, and Luke Zettlemoyer.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601–1611. Association for Computational Linguistics, 2017.doi: 10.18653/V1/P17-1147.URL https://doi.org/10.18653/v1/P17-1147.
Rajpurkar etal. [2018]Pranav Rajpurkar, Robin Jia, and Percy Liang.Know what you don’t know: Unanswerable questions for squad.In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784–789. Association for Computational Linguistics, 2018.doi: 10.18653/V1/P18-2124.URL https://aclanthology.org/P18-2124/.
Kwiatkowski etal. [2019]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, AnkurP. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, AndrewM. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.Natural questions: a benchmark for question answering research.Trans. Assoc. Comput. Linguistics, 7:452–466, 2019.doi: 10.1162/TACL\_A\_00276.URL https://doi.org/10.1162/tacl_a_00276.
Berant etal. [2013]Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.Semantic parsing on freebase from question-answer pairs.In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1533–1544. ACL, 2013.URL https://aclanthology.org/D13-1160/.
Clark etal. [2018]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.Think you have solved question answering? try arc, the AI2 reasoning challenge.CoRR, abs/1803.05457, 2018.URL http://arxiv.org/abs/1803.05457.
Zellers etal. [2019]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.Hellaswag: Can a machine really finish your sentence?In Anna Korhonen, DavidR. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019.doi: 10.18653/V1/P19-1472.URL https://doi.org/10.18653/v1/p19-1472.
Clark etal. [2019]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.Boolq: Exploring the surprising difficulty of natural yes/no questions.In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2924–2936. Association for Computational Linguistics, 2019.doi: 10.18653/V1/N19-1300.URL https://doi.org/10.18653/v1/n19-1300.
Bisk etal. [2020]Yonatan Bisk, Rowan Zellers, RonanLe Bras, Jianfeng Gao, and Yejin Choi.PIQA: reasoning about physical commonsense in natural language.In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020.doi: 10.1609/AAAI.V34I05.6239.URL https://doi.org/10.1609/aaai.v34i05.6239.
Sakaguchi etal. [2020]Keisuke Sakaguchi, RonanLe Bras, Chandra Bhagavatula, and Yejin Choi.Winogrande: An adversarial winograd schema challenge at scale.In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732–8740. AAAI Press, 2020.doi: 10.1609/AAAI.V34I05.6399.URL https://doi.org/10.1609/aaai.v34i05.6399.

Appendix A Broader Impact

We introduce a Coordinate Descent based approach, CDQuant, for compressing Large Language Models. Our method uses only a small amount of data for calibration. We do not foresee any ethical implications arising from the technical aspects of our approach. However, compressing LLMs may give rise to bias effects, a study of which seems essential given the extensive use of LLMs. Our work may be of assistance to such studies. Also, since quantization allows for easier deployment of LLMs, it could have potential societal implications which seem difficult to predict.

Appendix B Limitations

•
While both CD and BCD outperformed GPTQ in our experiments, BCD achieves slightly better performance than CD.However, BCD is not as fast as CD and can be expensive for large models. In future, we aim to develop techniques to speed up BCD.
•
Our algorithms still don’t bridge the gap between QAT and PTQ, especially on smaller models. To bridge this gap, we believe one should move away from the $\ell_{2}^{2}$ surrogate loss that is being considered by most of the existing work. Instead, we should design surrogate losses that are more closely aligned with end-to-end loss.
•
Due to computational reasons, we couldn’t run experiments where we replace GPTQ component with CDQuantin algorithms such as QuIP, FrameQuant.

Appendix C CDQuantfor sub-channel quantization

To simplify the explanation in this section, we introduce a slightly modified notation. Given a weight vector $\mathbf{w}$ , we represent it’s sub-channel quantization as $\mathbf{a}\odot\mathbf{q}+\mathbf{b}$ , where $\odot$ is the elementwise multiplication, and $\mathbf{a},\mathbf{b}\in\mathbb{R}^{d_{\text{in}}}$ are the scale, and bias parameters that satisfy the following constraints: $\mathbf{a}_{k}=\mathbf{a}_{l},\mathbf{b}_{k}=\mathbf{b}_{l}$ , for any two indices $k,l$ that fall in the same group. With this notation, for any given $\mathbf{a},\mathbf{b}$ , the optimization problem in Equation(3) can be rewritten as

\displaystyle\min_{\mathbf{q}}\|X(\mathbf{w}-\mathbf{a}\odot\mathbf{q}-\mathbf%{b})\|_{2}^{2}.

(5)

Letting $D_{\mathbf{a}}=\mbox{diag}(\mathbf{a}),\tilde{X}=XD_{\mathbf{a}},\tilde{%\mathbf{w}}=D_{\mathbf{a}}^{-1}\mathbf{w},\tilde{\mathbf{b}}=D_{\mathbf{a}}^{-%1}\mathbf{b}$ , the above problem can be further rewritten as

\displaystyle\min_{\mathbf{q}}\|\tilde{X}(\tilde{\mathbf{w}}-\mathbf{q}-\tilde%{\mathbf{b}})\|_{2}^{2}.

(6)

Observe that this problem is the same as the per-channel quantization problem described in Equation(1), but with modified parameters $\tilde{X},\tilde{\mathbf{w}},\tilde{\mathbf{b}}$ . So extending CDQuantto sub-channel quantization simply involves running Algorithms1,2 with these modified parameters. Algorithms4,5 present the resulting algorithms.

1:Input: $T$ - coordinate descent steps, $X$ - input data matrix, $\mathbf{w}$ - vector to be quantized, $a$ - scale, $b$ - bias, $\mathbf{q}_{0}$ - initial estimate

2: $\tilde{X}\leftarrow X\mbox{diag}(\mathbf{a}),\tilde{\mathbf{w}}=\mbox{diag}(%\mathbf{a})^{-1}\mathbf{w},\tilde{\mathbf{b}}=\mbox{diag}(\mathbf{a})\mathbf{b}$

3:Compute Hessian $H$ as: $H\leftarrow\tilde{X}^{T}\tilde{X}$

4:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow 2H(\mathbf{q}_{0}-(\tilde{\mathbf{w}}-\tilde{\mathbf{b}}))$

5:for $t\in[1:T]$ do

6:Find the coordinate that leads to the largest reduction in loss

i^{*},r^{*}=\operatorname*{arg\,min}_{i\in\{0,1,\dots d_{\text{in}}-1\},r\in\{%0,1,\dots 2^{c}-1\}}(r-\mathbf{q}_{t-1,i})^{2}H_{i,i}+(r-\mathbf{q}_{t-1,i})%\mathbf{g}_{i}

7:Update gradient $\mathbf{g}$ as

\mathbf{g}\leftarrow\mathbf{g}+2(r^{*}-\mathbf{q}_{t-1,i^{*}})H_{i^{*},\cdot},

where $H_{i^{*},\cdot}$ is the $i^{*}$ column of $H$

8:Update $\mathbf{q}_{t-1}$ as

\mathbf{q}_{t}\leftarrow\mathbf{q}_{t-1}+(r^{*}-\mathbf{q}_{t-1,i^{*}})\mathbf%{e}_{i^{*}},

where $\mathbf{e}_{i^{*}}$ is the standard basis vector with $1$ in $i^{*}$ position and $0$ everywhere else

9:endfor

1:Input: $T$ - coordinate descent steps, $k$ - block size, $X$ - input data matrix, $\mathbf{w}$ - vector to be quantized, $a$ - scale, $b$ - bias, $\mathbf{q}_{0}$ - initial estimate

2: $\tilde{X}\leftarrow X\mbox{diag}(\mathbf{a}),\tilde{\mathbf{w}}=\mbox{diag}(%\mathbf{a})^{-1}\mathbf{w},\tilde{\mathbf{b}}=\mbox{diag}(\mathbf{a})\mathbf{b}$

3:Compute Hessian $H$ as: $H\leftarrow\tilde{X}^{T}\tilde{X}$

4:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow 2H(\mathbf{q}_{0}-(\tilde{\mathbf{w}}-\tilde{\mathbf{b}}))$

5:for $t\in[1:T]$ do

6:Randomly partition the set $\{0,1,\dots d_{\text{in}}-1\}$ into $d_{\text{in}}/k$ blocks, each of size $k$

7:Find the block that leads to the largest reduction in loss

i^{*},r^{*}=\operatorname*{arg\,min}_{\begin{subarray}{c}i\in\{0,1,\dots d_{%\text{in}}/k-1\},\\r\in\{0,1,\dots 2^{c}-1\}^{k}\end{subarray}}(r-\mathbf{q}_{t-1,i})^{T}H_{i,i}(%r-\mathbf{q}_{t-1,i})+(r-\mathbf{q}_{t-1,i})^{T}\mathbf{g}_{i},

where $\mathbf{q}_{t-1,i},H_{i,i}$ are the sub-vector, sub-matrix of $\mathbf{q}_{t-1},H$ corresponding to block $i$ .

8:Update gradient $\mathbf{g}$ as

\mathbf{g}\leftarrow\mathbf{g}+2H_{i^{*},\cdot}(r^{*}-\mathbf{q}_{t-1,i^{*}}),

9:Update $\mathbf{q}_{t-1}$ as

\mathbf{q}_{t-1,i^{*}}\leftarrow r^{*},\quad\mathbf{q}_{t}\leftarrow\mathbf{q}%_{t-1},

10:endfor

Appendix D Detailed Downstream Evaluation Results

Config	Method	PaLM2-Bison
		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
w16a16	-	$27.84$	$75.19$	$77.42$	$23.87$	$59.04$	$84.64$	$88.87$	$82.50$	$82.43$	$78.22$
w3a16	OWC	$22.16$	$75.84$	$70.31$	$21.75$	$57.34$	$83.50$	$86.94$	$79.60$	$81.12$	$78.06$
w3a16	GPTQ	$24.29$	$74.65$	$72.29$	$22.98$	$58.70$	$85.02$	$87.58$	$80.81$	$81.72$	$77.66$
	CD	$23.60$	$74.92$	$72.25$	$23.13$	$58.53$	$84.68$	$88.20$	$80.60$	$81.61$	$77.74$
	BCD(k=2)	$23.24$	$74.78$	$72.06$	$22.24$	$58.62$	$84.55$	$88.04$	$80.69$	$81.83$	$78.22$
w3a16g128	OWC	$23.88$	$73.59$	$73.31$	$21.56$	$57.85$	$83.84$	$88.23$	$80.67$	$80.79$	$78.69$
w3a16g128	GPTQ	$25.32$	$73.96$	$74.10$	$23.23$	$58.02$	$84.39$	$88.50$	$80.67$	$81.83$	$78.06$
	CD	$25.24$	$74.16$	$74.11$	$23.18$	$57.51$	$83.75$	$88.38$	$81.15$	$82.15$	$77.98$
	BCD(k=2)	$24.93$	$74.05$	$73.90$	$23.18$	$56.74$	$84.05$	$88.26$	$81.06$	$82.43$	$78.53$
	OWC-CD	$24.60$	$74.03$	$73.80$	$21.75$	$57.17$	$84.18$	$88.2$	$80.97$	$81.94$	$77.27$
	OWC-CD+ CD	$25.12$	$74.37$	$73.99$	$23.38$	$58.36$	$85.44$	$88.65$	$80.93$	$81.77$	$77.98$
	OWC-CD+ BCD(k=2)	$25.32$	$73.54$	$74.30$	$23.18$	$57.34$	$83.96$	$88.59$	$81.01$	$81.39$	$77.51$
w4a16	OWC	$26.57$	$74.66$	$75.84$	$23.13$	$58.28$	$83.42$	$88.75$	$81.39$	$81.66$	$78.85$
w4a16	GPTQ	$27.06$	$74.87$	$76.45$	$22.74$	$58.87$	$84.60$	$88.53$	$81.72$	$82.48$	$78.69$
	CD	$26.79$	$74.40$	$76.35$	$22.83$	$59.22$	$84.68$	$88.99$	$81.60$	$82.10$	$78.69$
	BCD(k=2)	$27.06$	$74.54$	$76.28$	$23.13$	$59.81$	$84.76$	$88.56$	$81.70$	$82.26$	$77.74$
w4a16g128	OWC	$27.04$	$75.37$	$76.42$	$23.13$	$59.73$	$84.97$	$88.87$	$81.94$	$82.32$	$78.45$
w4a16g128	GPTQ	$27.12$	$75.63$	$76.42$	$23.38$	$58.45$	$84.76$	$89.05$	$82.01$	$82.15$	$78.85$
	CD	$27.31$	$75.52$	$76.45$	$23.62$	$59.04$	$85.02$	$88.50$	$82.04$	$82.54$	$78.69$
	BCD(k=2)	$27.37$	$75.44$	$76.92$	$23.62$	$58.87$	$85.35$	$88.72$	$81.98$	$82.97$	$79.48$
	OWC-CD	$27.31$	$75.70$	$76.64$	$22.93$	$58.36$	$84.64$	$88.56$	$81.98$	$82.59$	$78.14$
	OWC-CD+ CD	$27.01$	$75.52$	$76.69$	$23.28$	$58.79$	$85.10$	$88.56$	$81.98$	$82.15$	$78.69$
	OWC-CD+ BCD(k=2)	$26.98$	$75.42$	$76.84$	$23.33$	$58.79$	$85.35$	$88.47$	$81.93$	$82.32$	$78.14$

Config	Method	PaLM2-Otter
		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
w16a16	-	$19.78$	$70.3$	$69.1$	$18.06$	$54.1$	$80.13$	$85.47$	$79.51$	$79.43$	$73.95$
w3a16	OWC	$5.96$	$50.17$	$24.8$	$3.15$	$45.99$	$66.84$	$74.83$	$66.98$	$72.47$	$67.56$
w3a16	GPTQ	$13.77$	$66.12$	$56.15$	$12.99$	$52.56$	$78.58$	$85.17$	$76.75$	$78.35$	$72.3$
	CD	$14.49$	$65.71$	$58.83$	$13.44$	$52.47$	$79.92$	$83.03$	$75.81$	$77.09$	$71.9$
	BCD(k=2)	$14.63$	$65.96$	$59.46$	$14.07$	$52.73$	$79.84$	$83.09$	$75.93$	$77.42$	$71.74$
w3a16g128	OWC	$10.89$	$67.98$	$45.03$	$8.81$	$48.29$	$74.83$	$79.14$	$74.64$	$75.52$	$72.45$
w3a16g128	GPTQ	$16.12$	$69.8$	$60.82$	$14.47$	$53.16$	$79.34$	$84.07$	$77.12$	$77.86$	$73.16$
	CD	$15.46$	$71.62$	$62.07$	$16.19$	$52.3$	$78.79$	$84.34$	$77.02$	$78.35$	$74.82$
	BCD(k=2)	$16.04$	$71.04$	$62.02$	$15.65$	$51.71$	$78.32$	$84.83$	$77.04$	$77.69$	$74.03$
	OWC-CD	$15.62$	$71.78$	$61.19$	$14.76$	$52.99$	$79.34$	$84.1$	$76.92$	$77.42$	$75.22$
	OWC-CD+ CD	$16.2$	$70.77$	$61.8$	$14.71$	$53.41$	$79.42$	$84.5$	$77.22$	$78.29$	$73.64$
	OWC-CD+ BCD(k=2)	$16.7$	$71.3$	$61.87$	$14.91$	$53.84$	$79.8$	$84.37$	$77.25$	$78.45$	$74.19$
w4a16	OWC	$16.81$	$71.7$	$61.75$	$16.54$	$53.33$	$78.24$	$85.17$	$77.32$	$79.33$	$71.9$
w4a16	GPTQ	$19$	$70.66$	$66.01$	$16.34$	$53.67$	$79.46$	$85.57$	$78.5$	$79.05$	$72.93$
	CD	$18.7$	$70.97$	$66.11$	$17.08$	$54.27$	$80.89$	$85.87$	$78.55$	$79.05$	$72.38$
	BCD(k=2)	$19.14$	$70.9$	$65.72$	$16.88$	$54.78$	$80.6$	$85.6$	$78.64$	$79.65$	$73.72$
w4a16g128	OWC	$18.03$	$69.97$	$65.28$	$15.94$	$53.41$	$79.42$	$84.89$	$78.1$	$79.16$	$73.56$
w4a16g128	GPTQ	$19.17$	$70.34$	$67.46$	$17.18$	$53.58$	$79.88$	$85.02$	$79$	$79.11$	$73.56$
	CD	$19.25$	$71.12$	$67.61$	$17.13$	$53.33$	$80.18$	$84.92$	$79.09$	$79.11$	$72.93$
	BCD(k=2)	$19.45$	$70.81$	$67.75$	$17.08$	$54.27$	$80.3$	$84.74$	$79.05$	$78.94$	$72.93$
	OWC-CD	$19.2$	$70.58$	$67.66$	$17.22$	$53.24$	$80.51$	$85.05$	$78.84$	$79.49$	$74.74$
	OWC-CD+ CD	$19.64$	$70.8$	$68.02$	$17.52$	$53.75$	$80.77$	$84.98$	$79.09$	$78.78$	$74.27$
	OWC-CD+ BCD(k=2)	$19.31$	$70.41$	$67.91$	$17.37$	$53.67$	$80.3$	$84.95$	$79.07$	$78.78$	$74.43$

Config	Method	PaLM2-Gecko
		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
w16a16	-	$7.67$	$49.04$	$37.72$	$8.96$	$37.12$	$68.1$	$66.15$	$61.55$	$73.99$	$61.72$
w3a16	OWC	$3.71$	$48.17$	$18.84$	$5.07$	$30.03$	$60.1$	$59.69$	$53.45$	$70.24$	$60.93$
w3a16	GPTQ	$4.38$	$46.91$	$20.43$	$5.95$	$31.23$	$62.84$	$63.55$	$56.51$	$71.65$	$60.46$
	CD	$4.63$	$48.1$	$22.79$	$7.23$	$32.59$	$63.51$	$62.14$	$57.6$	$72.09$	$60.38$
	BCD(k=2)	$4.52$	$48.84$	$22.09$	$6.64$	$32.51$	$64.14$	$63.24$	$57.53$	$72.2$	$61.01$
w3a16g128	OWC	$4.43$	$57.48$	$20.99$	$8.42$	$32.59$	$63.09$	$62.05$	$53.45$	$71.38$	$59.59$
w3a16g128	GPTQ	$4.85$	$53.14$	$23.25$	$7.73$	$33.11$	$64.73$	$64.43$	$56.19$	$72.42$	$61.09$
	CD	$4.49$	$54.43$	$24.12$	$8.12$	$33.28$	$65.15$	$64.22$	$56.56$	$71.98$	$61.64$
	BCD(k=2)	$4.29$	$53.58$	$24.41$	$8.27$	$33.7$	$65.32$	$64.25$	$56.25$	$71.71$	$61.88$
	OWC-CD	$4.6$	$53.86$	$23.23$	$7.14$	$32.85$	$64.81$	$64.65$	$56.31$	$71.82$	$61.72$
	OWC-CD+ CD	$4.52$	$52.4$	$25.07$	$7.78$	$32.76$	$64.44$	$65.11$	$56.85$	$71.93$	$61.96$
	OWC-CD+ BCD(k=2)	$4.65$	$51.15$	$25.13$	$7.68$	$33.19$	$64.9$	$64.68$	$56.88$	$71.82$	$61.17$
w4a16	OWC	$6.09$	$48.4$	$31.06$	$7.53$	$35.24$	$65.07$	$64.59$	$59.59$	$72.96$	$63.85$
w4a16	GPTQ	$6.81$	$50.67$	$32.5$	$8.07$	$36.43$	$65.53$	$65.05$	$60.16$	$73.29$	$63.85$
	CD	$6.81$	$51.07$	$31.87$	$8.66$	$35.58$	$66.92$	$64.65$	$60.22$	$73.23$	$63.14$
	BCD(k=2)	$6.65$	$50.86$	$31.67$	$8.91$	$35.15$	$67.21$	$64.4$	$60.31$	$73.45$	$63.22$
w4a16g128	OWC	$7.12$	$47.85$	$33.58$	$9.99$	$35.58$	$67.09$	$68.38$	$60.15$	$73.5$	$62.9$
w4a16g128	GPTQ	$7.2$	$51.6$	$33.94$	$8.86$	$36.18$	$67.21$	$66.51$	$59.99$	$73.5$	$63.38$
	CD	$7.59$	$48.87$	$35.22$	$9.06$	$35.92$	$68.01$	$67.09$	$60.49$	$73.45$	$63.54$
	BCD(k=2)	$7.29$	$48.72$	$34.67$	$9.1$	$36.69$	$68.01$	$67.28$	$60.7$	$73.34$	$62.9$
	OWC-CD	$7.31$	$48.52$	$33.2$	$9.3$	$35.75$	$67.59$	$66.97$	$60.8$	$74.21$	$63.06$
	OWC-CD+ CD	$7.15$	$49.15$	$34.77$	$9.4$	$36.52$	$67.8$	$66.33$	$60.74$	$73.61$	$63.61$
	OWC-CD+ BCD(k=2)	$7.29$	$48.88$	$34.68$	$9.15$	$36.43$	$68.01$	$65.75$	$60.77$	$74.05$	$64.01$

Method	Epochs	PaLM2-Otter
w2a16g128		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
GPTQ	-	$6.93$	$64.89$	$28.36$	$7.73$	$44.28$	$71.04$	$78.81$	$68.29$	$72.69$	$67.72$
CD	$1$	$8.23$	$68.12$	$32.98$	$10.14$	$43.69$	$72.43$	$76.91$	$68.11$	$72.96$	$66.46$
BCD(k=2)	$1$	$8.09$	$69.2$	$32.85$	$9.99$	$43.43$	$73.02$	$77$	$68.03$	$73.07$	$66.38$
BCD(k=2)	$2$	$8.2$	$69.43$	$32.98$	$9.74$	$43.52$	$73.7$	$77.03$	$68.56$	$74.1$	$67.8$
	$3$	$8.31$	$69.79$	$33.03$	$9.65$	$43.52$	$73.11$	$77.09$	$68.76$	$73.07$	$67.01$
BCD(k=4)	$1$	$7.87$	$69.52$	$33.24$	$9.79$	$43.17$	$72.6$	$77.83$	$68.61$	$73.88$	$66.46$
BCD(k=4)	$2$	$8.31$	$69.06$	$33.03$	$9.6$	$43$	$72.6$	$76.64$	$68.51$	$73.18$	$66.3$
	$3$	$8.31$	$69.8$	$32.69$	$9.5$	$42.15$	$72.43$	$77.06$	$68.34$	$73.72$	$67.88$

Method	Epochs	PaLM2-Bison
w2a16g128		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
GPTQ	-	$15.26$	$69.34$	$55.6$	$17.08$	$51.02$	$80.13$	$86.02$	$74.44$	$78.78$	$74.74$
CD	$1$	$15.6$	$67.43$	$55.06$	$16.83$	$50.43$	$79.97$	$83.76$	$74.35$	$78.35$	$74.51$
BCD(k=2)	$1$	$15.62$	$67.32$	$55.82$	$16.63$	$51.45$	$79.97$	$83.61$	$74.46$	$79$	$75.45$
BCD(k=2)	$2$	$15.62$	$68.45$	$56.36$	$16.98$	$51.02$	$80.09$	$84.56$	$74.99$	$78.89$	$75.06$
	$3$	$16.01$	$67.99$	$56.31$	$16.88$	$51.19$	$79.88$	$84.65$	$74.95$	$79.33$	$75.45$
BCD(k=4)	$1$	$15.9$	$67.46$	$56.12$	$16.44$	$51.28$	$80.35$	$84.01$	$74.89$	$78.84$	$75.85$
BCD(k=4)	$2$	$16.04$	$67.61$	$56.34$	$16.39$	$51.54$	$80.09$	$83.91$	$74.83$	$78.73$	$75.37$
	$3$	$16.12$	$67.54$	$56.3$	$16.24$	$51.02$	$80.22$	$83.98$	$74.93$	$78.51$	$75.93$

Config	Method	PaLM2-Gecko			PaLM2-Otter			PaLM2-Bison
		Gen.	Rank	Avg.	Gen.	Rank	Avg.	Gen.	Rank	Avg.
w16a16	-	$25.85$	$61.44$	$47.2$	$44.31$	$75.43$	$62.98$	$51.08$	$78.28$	$68.00$
w3a16	OWC	$15.45$	$55.73$	$39.62$	$15.06$	$59.29$	$41.60$	$39.58$	$75.45$	$61.10$
w3a16	GPTQ	$13.08$	$56.92$	$39.39$	$31.80$	$71.94$	$55.88$	$41.42$	$76.70$	$62.58$
	CD	$15.82$	$58.19$	$41.24$	$32.34$	$72.03$	$56.16$	$41.66$	$76.71$	$62.69$
	BCD(k=2)	$16.16$	$57.85$	$41.17$	$32.04$	$72.22$	$56.15$	$41.37$	$76.78$	$62.62$
w3a16g128	OWC	$18.54$	$56.95$	$41.59$	$26.62$	$68.09$	$51.50$	$41.54$	$76.51$	$62.52$
w3a16g128	GPTQ	$17.96$	$58.30$	$42.16$	$33.43$	$72.98$	$57.16$	$43.47$	$76.65$	$63.38$
	CD	$16.81$	$58.10$	$41.58$	$32.73$	$72.62$	$56.66$	$42.95$	$77.03$	$63.4$
	BCD(k=2)	$16.33$	$57.81$	$41.22$	$32.13$	$72.67$	$56.46$	$42.67$	$76.85$	$63.18$
	OWC-CD	$16.73$	$57.74$	$41.34$	$32.90$	$72.40$	$56.60$	$42.23$	$77.02$	$63.11$
	OWC-CD+ CD	$16.37$	$58.22$	$41.48$	$33.53$	$72.64$	$56.99$	$43.09$	$76.91$	$63.38$
	OWC-CD+ BCD(k=2)	$16.47$	$58.45$	$41.66$	$33.71$	$72.73$	$57.12$	$43.07$	$76.70$	$63.25$
w4a16	OWC	$16.07$	$59.42$	$42.08$	$33.76$	$72.22$	$56.83$	$43.59$	$77.13$	$63.71$
w4a16	GPTQ	$18.77$	$59.5$	$43.21$	$34.03$	$73.53$	$57.73$	$44.16$	$77.54$	$64.19$
	CD	$18.20$	$59.95$	$43.25$	$35.01$	$73.42$	$58.06$	$43.97$	$77.55$	$64.12$
	BCD(k=2)	$17.89$	$59.64$	$42.94$	$35.10$	$73.31$	$58.02$	$43.95$	$77.77$	$64.24$
w4a16g128	OWC	$19.00$	$60.42$	$43.85$	$34.52$	$72.83$	$57.5$	$44.45$	$77.71$	$64.41$
w4a16g128	GPTQ	$19.07$	$60.40$	$43.87$	$35.34$	$73.60$	$58.29$	$44.25$	$77.65$	$64.29$
	CD	$20.35$	$60.94$	$44.70$	$36.20$	$73.67$	$58.68$	$44.21$	$77.72$	$64.31$
	BCD(k=2)	$20.44$	$60.69$	$44.59$	$36.11$	$73.54$	$58.57$	$44.17$	$77.56$	$64.2$
	OWC-CD	$19.80$	$60.39$	$44.15$	$36.33$	$73.62$	$58.71$	$44.75$	$77.70$	$64.52$
	OWC-CD+ CD	$20.47$	$60.85$	$44.70$	$35.98$	$73.59$	$58.54$	$44.77$	$77.52$	$64.42$
	OWC-CD+ BCD(k=2)	$20.15$	$60.49$	$44.36$	$36.00$	$73.62$	$58.57$	$44.70$	$77.70$	$64.50$

Config	Method	PaLM2-Bison
		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
w16a16	-	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$
w3a16	OWC	$15.87$	$73.01$	$58.86$	$10.58$	$52.47$	$76.43$	$86.91$	$79.76$	$81.01$	$76.09$
w3a16	GPTQ	$17.31$	$71.25$	$65.53$	$11.56$	$54.44$	$79.25$	$87.22$	$80.78$	$81.61$	$76.87$
	CD	$17.84$	$71.24$	$64.76$	$12.8$	$54.44$	$79.17$	$87.58$	$80.79$	$82.1$	$76.16$
	BCD(k=2)	$17.81$	$71.3$	$64.29$	$12.06$	$53.5$	$79.08$	$87.65$	$80.91$	$82.21$	$77.35$
w3a16g128	OWC	$15.98$	$71.28$	$68.02$	$10.88$	$53.58$	$79.55$	$86.73$	$80.9$	$81.61$	$76.72$
w3a16g128	GPTQ	$18.95$	$71.49$	$69.67$	$13.78$	$53.24$	$79.84$	$86.79$	$81.05$	$82.05$	$76.95$
	CD	$18.31$	$71.81$	$68.56$	$13.14$	$53.41$	$79.8$	$87.55$	$81.31$	$82.43$	$77.66$
	BCD(k=2)	$18.53$	$71.55$	$67.62$	$12.99$	$53.58$	$79.59$	$87.49$	$81.14$	$82.81$	$76.48$
	OWC-CD	$17.26$	$71.79$	$67.53$	$12.35$	$54.69$	$80.85$	$85.72$	$81.02$	$81.88$	$77.98$
	OWC-CD+ CD	$18.75$	$72.05$	$68.35$	$13.19$	$54.1$	$80.18$	$86.36$	$80.86$	$82.48$	$77.51$
	OWC-CD+ BCD(k=2)	$19.36$	$71.47$	$68.45$	$12.99$	$53.58$	$80.47$	$86.02$	$81.15$	$82.26$	$76.72$
w4a16	OWC	$20$	$72.51$	$69.34$	$12.5$	$54.61$	$80.39$	$87.06$	$81.5$	$82.05$	$77.19$
w4a16	GPTQ	$21.25$	$72.39$	$70.6$	$12.4$	$55.2$	$80.64$	$87.58$	$81.98$	$82.26$	$77.58$
	CD	$21.5$	$71.57$	$70.04$	$12.8$	$55.29$	$80.56$	$88.07$	$81.78$	$82.59$	$77.03$
	BCD(k=2)	$21.16$	$71.92$	$70.01$	$12.7$	$55.63$	$81.19$	$87.77$	$81.82$	$82.81$	$77.43$
w4a16g128	OWC	$21.22$	$73.06$	$70.07$	$13.44$	$55.8$	$80.47$	$87.77$	$82.08$	$82.64$	$77.51$
w4a16g128	GPTQ	$20.72$	$73.17$	$70.12$	$12.99$	$55.38$	$80.89$	$87.55$	$82.14$	$82.59$	$77.35$
	CD	$21.8$	$73.37$	$69.06$	$12.6$	$54.78$	$80.72$	$87.25$	$82.16$	$83.19$	$78.22$
	BCD(k=2)	$21.5$	$73.06$	$69.34$	$12.8$	$55.29$	$80.98$	$87.31$	$82.14$	$82.54$	$77.11$
	OWC-CD	$21.36$	$73.85$	$70.07$	$13.73$	$55.12$	$81.23$	$87.71$	$82.27$	$82.59$	$77.27$
	OWC-CD+ CD	$21.41$	$73.32$	$71.46$	$12.89$	$54.18$	$81.4$	$87.28$	$82.17$	$82.37$	$77.74$
	OWC-CD+ BCD(k=2)	$21.19$	$73.24$	$71.64$	$12.75$	$55.03$	$81.52$	$87.28$	$82.06$	$82.64$	$77.66$

Config	Method	PaLM2-Otter
		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
w16a16	-	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$
w3a16	OWC	$2.27$	$49.45$	$8.02$	$0.49$	$41.98$	$60.14$	$48.78$	$66.91$	$71.87$	$66.06$
w3a16	GPTQ	$9.09$	$64.53$	$46.38$	$7.19$	$49.83$	$73.99$	$81.96$	$76.87$	$78.89$	$70.09$
	CD	$10.72$	$64.01$	$48.03$	$6.59$	$50.77$	$76.6$	$81.01$	$75.77$	$77.97$	$70.09$
	BCD(k=2)	$10.55$	$64.31$	$47.25$	$6.05$	$50.85$	$76.43$	$81.41$	$75.91$	$78.45$	$70.24$
w3a16g128	OWC	$6.18$	$66.37$	$30.48$	$3.44$	$47.44$	$70.16$	$69.63$	$74.61$	$76.88$	$69.85$
w3a16g128	GPTQ	$10.17$	$66.53$	$49.29$	$7.73$	$51.37$	$75.93$	$82.35$	$77$	$79.11$	$72.14$
	CD	$7.17$	$68.73$	$49.42$	$5.61$	$50.26$	$76.18$	$82.39$	$76.79$	$78.13$	$71.98$
	BCD(k=2)	$6.68$	$68.18$	$48.39$	$5.27$	$50.26$	$75.93$	$81.41$	$76.84$	$79$	$72.61$
	OWC-CD	$8.98$	$68.37$	$48.33$	$5.91$	$50.51$	$75.46$	$81.35$	$76.83$	$78.18$	$72.06$
	OWC-CD+ CD	$9.72$	$67.87$	$49.87$	$6.64$	$51.37$	$76.68$	$81.59$	$76.78$	$77.91$	$71.51$
	OWC-CD+ BCD(k=2)	$9.94$	$68.26$	$49.91$	$6.74$	$50.85$	$76.22$	$81.68$	$76.99$	$78.89$	$71.74$
w4a16	OWC	$11.19$	$66.67$	$49.56$	$7.63$	$50$	$73.91$	$80$	$77.36$	$79.11$	$72.93$
w4a16	GPTQ	$12.27$	$66.65$	$48.72$	$8.46$	$51.71$	$75.84$	$82.02$	$78.2$	$80.09$	$73.32$
	CD	$12.52$	$66.85$	$53.16$	$7.53$	$52.3$	$76.43$	$81.28$	$78.3$	$79.6$	$72.61$
	BCD(k=2)	$12.33$	$67$	$53.45$	$7.63$	$51.96$	$75.97$	$80.92$	$78.47$	$80.47$	$72.06$
w4a16g128	OWC	$11.33$	$65.98$	$53.57$	$7.19$	$49.66$	$74.92$	$82.17$	$78.22$	$79.54$	$72.45$
w4a16g128	GPTQ	$11.5$	$65.81$	$56.36$	$7.68$	$51.71$	$76.52$	$82.08$	$78.86$	$79.27$	$73.16$
	CD	$11.72$	$66.85$	$57.63$	$8.61$	$51.19$	$76.6$	$82.51$	$78.89$	$79.49$	$73.32$
	BCD(k=2)	$11.66$	$66.73$	$57.5$	$8.56$	$51.19$	$76.3$	$82.29$	$78.88$	$79.27$	$73.32$
	OWC-CD	$12.27$	$66.28$	$58.21$	$8.56$	$51.62$	$76.85$	$82.32$	$78.85$	$79.49$	$72.61$
	OWC-CD+ CD	$11.63$	$66.28$	$58.06$	$7.92$	$51.62$	$76.94$	$82.45$	$78.72$	$79.11$	$72.69$
	OWC-CD+ BCD(k=2)	$11.72$	$65.96$	$58.29$	$8.02$	$51.79$	$76.77$	$82.17$	$78.8$	$79.49$	$72.69$

Config	Method	PaLM2-Gecko
		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
w16a16	-	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$	$aa$
w3a16	OWC	$2.55$	$47.57$	$9.81$	$1.87$	$29.69$	$56.73$	$64.8$	$53.28$	$69.59$	$60.3$
w3a16	GPTQ	$3.35$	$44.13$	$3.97$	$0.89$	$30.2$	$62.12$	$63.79$	$56.33$	$71.22$	$57.85$
	CD	$3.55$	$45.71$	$11.32$	$2.71$	$30.63$	$60.56$	$66.45$	$57.2$	$71.87$	$62.43$
	BCD(k=2)	$3.63$	$47.04$	$11.22$	$2.76$	$30.29$	$60.52$	$65.32$	$57.33$	$71.82$	$61.8$
w3a16g128	OWC	$3.68$	$55.38$	$12.5$	$2.61$	$31.66$	$60.14$	$66.12$	$53.19$	$71.71$	$58.88$
w3a16g128	GPTQ	$4.02$	$50.04$	$14.64$	$3.15$	$32.42$	$61.91$	$66.64$	$56.05$	$73.01$	$59.75$
	CD	$3.38$	$52.01$	$9.62$	$2.21$	$31.23$	$63.13$	$65.9$	$55.98$	$72.47$	$59.91$
	BCD(k=2)	$3.1$	$50.95$	$9.36$	$1.92$	$31.14$	$62.25$	$65.41$	$56.16$	$72.91$	$58.96$
	OWC-CD	$3.6$	$51.69$	$9.33$	$2.31$	$31.83$	$61.74$	$65.02$	$56.23$	$72.2$	$59.43$
	OWC-CD+ CD	$3.82$	$50.39$	$9.5$	$1.77$	$32.51$	$62.25$	$66.33$	$56.91$	$71.27$	$60.06$
	OWC-CD+ BCD(k=2)	$3.63$	$49.87$	$10.26$	$2.12$	$32.76$	$62.88$	$66.24$	$56.79$	$71.82$	$60.22$
w4a16	OWC	$4.02$	$46.96$	$10.63$	$2.66$	$33.19$	$61.99$	$66.7$	$59.26$	$73.67$	$61.72$
w4a16	GPTQ	$4.46$	$48.69$	$18.84$	$3.1$	$34.47$	$62.96$	$64.77$	$60.09$	$73.61$	$61.09$
	CD	$4.13$	$48.88$	$16.64$	$3.15$	$33.53$	$63.89$	$66.79$	$60.31$	$73.78$	$61.4$
	BCD(k=2)	$4.24$	$48.31$	$15.73$	$3.3$	$33.28$	$63.34$	$66.39$	$60.26$	$73.56$	$61.01$
w4a16g128	OWC	$4.99$	$46.1$	$20.62$	$4.28$	$34.73$	$64.48$	$67.83$	$59.91$	$74.32$	$61.25$
w4a16g128	GPTQ	$5.18$	$49.79$	$18.17$	$3.15$	$35.15$	$64.6$	$66.24$	$60.21$	$74.32$	$61.88$
	CD	$5.73$	$46.96$	$24.17$	$4.53$	$36.18$	$66.04$	$67.8$	$60.7$	$73.67$	$61.25$
	BCD(k=2)	$5.79$	$47.28$	$24.11$	$4.58$	$36.01$	$65.7$	$67.06$	$60.73$	$73.61$	$61.01$
	OWC-CD	$5.54$	$47.24$	$21.71$	$4.72$	$34.3$	$64.73$	$68.01$	$60.72$	$73.56$	$61.01$
	OWC-CD+ CD	$5.62$	$47.6$	$24$	$4.68$	$35.49$	$66.16$	$67.06$	$60.78$	$74.1$	$61.48$
	OWC-CD+ BCD(k=2)	$5.57$	$47.17$	$23.05$	$4.82$	$35.24$	$65.82$	$66.02$	$60.9$	$73.23$	$61.72$

Method	Epochs	PaLM2-Otter			PaLM2-Bison
w2a16g128		Generation	Rank	Avg.	Generation	Rank	Avg.
GPTQ	-	$23.06$	$66.62$	$49.2$	$33.62$	$72.74$	$57.09$
CD	1	$25.12$	$65.73$	$49.48$	$33.63$	$71.80$	$56.53$
BCD(k=2)	$1$	$25.42$	$65.57$	$49.51$	$33.97$	$72.16$	$56.88$
BCD(k=2)	$2$	$24.73$	$65.78$	$49.36$	$34.01$	$72.39$	$57.04$
	$3$	$25.28$	$66.01$	$49.72$	$34.27$	$72.67$	$57.31$
BCD(k=4)	$1$	$25.15$	$66.04$	$49.68$	$33.94$	$72.10$	$56.84$
BCD(k=4)	$2$	$24.86$	$65.71$	$49.37$	$33.87$	$71.97$	$56.73$
	$3$	$25.70$	$65.30$	$49.46$	$33.85$	$72.06$	$56.78$

Method	Epochs	PaLM2-Otter
w2a16g128		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
GPTQ	-	$4.32$	$63.43$	$22.08$	$2.41$	$42.32$	$66.96$	$80.09$	$68.54$	$73.78$	$68.03$
CD	$1$	$4.99$	$66.51$	$25.53$	$3.44$	$40.87$	$70.33$	$74.89$	$68.13$	$72.96$	$67.17$
BCD(k=2)	$1$	$4.93$	$67.51$	$25.56$	$3.69$	$41.47$	$69.7$	$74.98$	$67.97$	$73.72$	$65.59$
BCD(k=2)	$2$	$4.6$	$67.74$	$23.6$	$3$	$40.78$	$69.49$	$75.47$	$68.27$	$73.39$	$67.25$
	$3$	$5.1$	$67.91$	$24.53$	$3.59$	$42.41$	$69.87$	$75.02$	$68.52$	$73.88$	$66.38$
BCD(k=4)	$1$	$4.9$	$67.5$	$25.01$	$3.2$	$42.15$	$68.69$	$76.18$	$68.51$	$73.61$	$67.09$
BCD(k=4)	$2$	$4.79$	$67.9$	$23.55$	$3.2$	$41.81$	$69.82$	$73.39$	$68.5$	$73.5$	$67.25$
	$3$	$5.46$	$68.29$	$26$	$3.05$	$41.21$	$69.02$	$73.98$	$68.06$	$72.52$	$67.01$

Method	Epochs	PaLM2-Bison
w2a16g128		NatualQ.	SQuAD	TriviaQA	WebQ	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	WinoGrande
GPTQ	-	$11.55$	$66.72$	$48.79$	$7.43$	$47.27$	$76.89$	$84.56$	$74.36$	$79.54$	$73.8$
CD	$1$	$11.25$	$64.49$	$50.84$	$7.92$	$48.12$	$75.97$	$79.39$	$74.28$	$79.11$	$73.95$
BCD(k=2)	$1$	$11.77$	$64.17$	$51.36$	$8.56$	$48.38$	$75.88$	$81.01$	$74.6$	$78.84$	$74.27$
BCD(k=2)	$2$	$11.61$	$64.69$	$51.23$	$8.51$	$47.87$	$76.09$	$81.74$	$74.72$	$79$	$74.9$
	$3$	$11.72$	$64.5$	$52.15$	$8.71$	$48.46$	$76.18$	$82.54$	$74.82$	$79.6$	$74.43$
BCD(k=4)	$1$	$11.69$	$64.48$	$51.07$	$8.51$	$47.7$	$75.63$	$80.89$	$74.55$	$79.6$	$74.27$
BCD(k=4)	$2$	$11.75$	$64.39$	$51.36$	$7.97$	$48.46$	$75.04$	$80.58$	$74.48$	$79.38$	$73.88$
	$3$	$11.8$	$64.41$	$51.16$	$8.02$	$48.12$	$75.55$	$80.86$	$74.55$	$78.73$	$74.59$

Appendix E Quantization Runtime

Config

Method

PaLM2-Otter

PaLM2-Bison

FFN runtime

(in minutes)

Attn. runtime

(in minutes)

FFN runtime

(in minutes)

Attn. runtime

(in minutes)

w3a16

GPTQ

0.90

0.72

6.42

0.63

2.94

0.49

16.13

0.43

BCD(k=2)

14.03

2.08

82.13

1.81

w3a16g128

GPTQ

1.02

0.75

6.70

0.66

3.32

0.59

18.44

0.52

BCD(k=2)

14.73

2.12

88.10

1.86

w4a16

GPTQ

0.90

0.64

6.60

0.56

3.59

0.59

21.19

0.52

BCD(k=2)

34.85

4.72

235.14

4.13

w4a16g128

GPTQ

0.98

0.69

6.962

0.60

3.99

0.69

23.48

0.60

BCD(k=2)

42.37

5.41

192.59

4.74

Appendix F Comparison to QuantEase

We also compare our per-channel quantization results with those of QuantEase, a parallel study to ours. It’s important to note that QuantEase does not have a publicly available implementation. So, we implemented the cyclic coordinate descent strategy used by QuantEase, and used the same initialization and regularization strength as our algorithms (although the QuantEase paper doesn’t provide these details). We then ran QuantEase for the recommended number of iterations specified in the paper ( $20$ epochs or $T=20d_{\text{in}}$ iterations). The findings are presented in Table20. On PaLM2-Gecko, PaLM2-Otter, and PaLM2-Bison, both these approaches have similar performance. These results collectively highlight the potential of coordinate descent algorithms in outperforming GPTQ.

Config	Method	PaLM2-Gecko	PaLM2-Otter	PaLM2-Bison
w3a16	GPTQ	$11.347$	$7.176$	$5.774$
	QuantEase (epochs=20)	$10.731$	$6.996$	$5.741$
	CD	$10.920$	$7.002$	$5.739$
	BCD(k=2)	$10.898$	$6.979$	$5.733$
w4a16	GPTQ	$8.764$	$6.249$	$5.417$
	QuantEase (epochs=20)	$8.670$	$6.197$	$5.408$
	CD	$8.694$	$6.195$	$5.407$
	BCD(k=2)	$8.691$	$6.192$	$5.405$

CDQuant: Accurate Post-training Weight Quantization of Large Pre-trained Models using Greedy Coordinate Descent (2024)

Abstract

1 Introduction

2 Related Work

Weight-only Quantization.

Weight+Activation Quantization.

Efficient End-to-End Quantization.

3 CDQuant

Notation.

3.1 Greedy Coordinate Descent

Extension to Block Coordinate Descent.

Initializing a,b,𝐪𝑎𝑏𝐪a,b,\mathbf{q}italic_a , italic_b , bold_q.

3.2 Extension to Sub-channel Quantization

Initialization.

4 Experiments

Quantization.

Models.

Baselines.

CDQuant.

Evaluation.

Training.

Results.

Runtime.

5 Conclusion and Future Work

Acknowledgements

References

Appendix A Broader Impact

Appendix B Limitations

Appendix C CDQuantfor sub-channel quantization

Appendix D Detailed Downstream Evaluation Results

Appendix E Quantization Runtime

Appendix F Comparison to QuantEase

Initializing $a,b,\mathbf{q}$ .