\section{Testing}\label{sec:Testing}

By using a combination of black box and white box
testing techniques, we have produced a comprehensive
set of tests that we believe would prove extensive for
any implementation of a routine designed to compute the
$l_2$ norm of a real or complex vector. 
% Needs to be run with gfortran and coverage
% to check out this claim
%The full test
%suite provides 100\% block and decision coverage for all
%(CHECK!!) the following implementations
All the following implementations have been tested using the complete suite
\begin{enumerate}
\item the original Level 1 Blas reference routines~\cite{acmalg539},

\item the reference LAPACK~\cite{Anderson:1999:LUG} implementation,

\item Kahan's algorithm in stand-alone mode,

%\item the \FaithfulNorm~\cite{Graillat:2015:ECF} implementation,

\item our proposed hybrid implementation.
\end{enumerate}

The tests have been devised to make the computed
results automatically checkable. 

The tests given below are all presented as real data. The
same data is used for testing the complex routines after
transforming it as follows:

\begin{description}
\item [$incx = 1$]
If the number of elements, $n$, in the real
input vector, $x$, is even, then we generate a
complex input vector, $xc$, of length $n/2$
where $xc(j) = CMPLX(x(2*j-1), x(2*j))$ for 
$j=1,\ldots,n/2$. If
$n$ is odd then the final element is $xc(\lceil n/2 \rceil)
=CMPLX(x(n), zero)$.

\item [$incx > 1$]
Here we generate a complex input vector, $xc$, of the
same length as the real input vector, $n$, with
$xc(i)$ set to  $CMPLX(x(i), zero)$ or
$CMPLX(zero, x(i))$ depending on whether  $i$
is odd or even respectively.

\end{description}

Using this transformation the results for the real and
complex data should be identical.

Table~\ref{tab:kahanieee} gives values for the parameters
used later in the paper to define some of the test data
sets. Numerical values are given for both single and double
precision IEEE standard formats.

The IEEE floating point status flags signalling on return from
any subprogram call must be the union of the flags signalling on
entry to the subprogram and those set within the routine itself.
When the input vector consists entirely of finite floating-point
values, 
the only flags that may be set as a result of a call to a \nrm
routine are overflow and inexact.
The overflow flag should then only signal when the
value of the final result is not representable in the precision
being used; in this case the inexact flag should also signal. If
one of the elements of the input vector is $\pm Inf$ then the
result is exactly $\pm Inf$ and no overflow flag is set. The
invalid flag will signal if a NaN value (but no Inf values)
are detected in the input vector.

Note that the underflow condition should never occur from a
call to a \nrm routine since $||x||_2 \ge |x_i| > 0$ provided
the vector, $x$, consists entirely of finite floating-point
values with at least one element non-zero. Thus, while an
intermediate computation could underflow if, for example,
no scaling were employed, the final correct result can only
be zero if all the elements of the vector are zero.

Any test failures that involve the inexact flag (element 5 in
the output vector of status flags) need to be considered carefully as
the result may vary depending upon the algorithm used. Many
of the test vectors contain elements that are either small
integers or small integers scaled by a power of two with a
resultant $l_2$ norm that is exactly representable in IEEE
floating point format. In these cases we do not expect
the inexact status flag to be raised. This is in line with
the goal we set ourselves in~\cite{Hanson:2015:RAC} although
it does favour algorithms that use scaling factors that are
powers of two since division and multiplication by such values
should not generate any rounding errors.

However some test cases may or may not signal inexact depending
on how the calculations are performed. One example of this is
discussed below for vectors of length one. A second more general example
is the vector $x= \left \{ 9*p, 11*p, 13*p, 23*p \right \}$
where $p$ is a power of two which is large enough to cause
the implementation to scale. Provided the result does not
overflow (i.e., $p$ is small enough) we would expect that the
exact result, $30*p$, will be returned. However, if any of the
chosen scale factors is not a power of two then the resultant
scaling/rescaling will generate rounding errors and an inexact
result will ensue. This will happen using both the original
BLAS implementation and the Lapack routine; details of both 
routines may be found in~\cite{Hanson:2015:RAC}).

When the $l_2$ norm cannot be represented exactly using IEEE
floating point format, the mathematically correct result
generally involves the square root of an integer value
multiplied by a power of two. When computing relative errors
we assume that the Fortran intrinsic function, SQRT, returns
a correctly rounded result which may be used to generate the
best available approximation to the exact value. Deviations
from this value then signal additional rounding errors which
may be due, for example, to poor scaling.

\begin{table}[tb]
\caption{Values of the constants used in setting test data values.}
\label{tab:kahanieee}
\begin{center}
\begin{tabular}{|c | c | c |}
\hline
  \mchead{1}{}    &  \mchead{1}{Single Precision\T\B}     &  \mchead{1}{Double Precision} \\
\hline
$    \maxreal$               
               & $(2-2^{-23})2^{127} \approx 2^{128}\T$  
               &  $(2-2^{-52})2^{1023} \approx 2^{1024}$   \\
$    \Tiny  $                  & $2^{-126}$  &  $2^{-1022}$  \\
$    \macheps$                 & $2^{-23}$   &  $2^{-52}$    \\
$    \Omega  $                 & $ 128   $   &  $ 1024 $     \\
$    \eta    $                 & $ -23   $   &  $ -52 $      \\
$    \halfeps  $               & $ -11   $   &  $ -26 $      \\
$    \minExponent$             & $-125$      &   $-1021$     \\
$    \digits $                 & $24$        &   $53$        \\
\hline
\mchead{3}{Key\T\B}\\
\hline
$\maxreal$ & \mchead{2}{HUGE(kind)\T} \\
$\Tiny$   & \mchead{2}{TINY(kind)} \\
$\macheps$ & \mchead{2}{EPSILON(kind)}  \\
$\Omega$   & \mchead{2}{EXPONENT(HUGE(kind))+1}\\
$\eta$   & \mchead{2}{EXPONENT($\epsilon$)-1}\\
$\halfeps$   & \mchead{2}{$\lfloor\epsilon/2\rfloor$}\\
$\minExponent$ & \mchead{2}{MINEXPONENT(kind)} \\
$\digits$   & \mchead{2}{DIGITS(kind)\B} \\
\hline
\mchead{3}{\textit{NOTE: the definition of $\maxreal$ and $\Tiny$ used here }\T}\\
\mchead{3}{\textit{differ from those given in the main article.}\B}\\
\hline
\end{tabular}
\end{center}
\end{table}

\subsection{Tests for \bignrm Routines}

We can split the input domain
into a number of `obvious' classes
\begin{enumerate}
\item\label{test:rep} Inputs which generate a representable
floating point value,
\item\label{test:nonrep} Inputs which generate a
non-representable floating-point value,
\item\label{test:nonfinite} Input data that includes non-finite
values,
\item\label{test:nans} Input data that includes not-a-number
values,
\item\label{test:trivial} Trivial input data,
\item\label{test:illegal} Illegal input data.
\end{enumerate}

We start with classes~\ref{test:trivial}
and~\ref{test:illegal} since these are small classes. As we
are aiming to deal with any data within the input vector,
i.e., both valid floating-point numbers as well as
infinite and NaN values,
the only illegal data is obtained by setting \fvarn{incx} to be
less than one. The only trivial data is the case of a null array
which we assume is being signaled if the value of \fvarn{n} is
set to be non-positive. In both cases we choose to return the
value zero for the norm in line with previous implementations.
Thus our first two sets of test data are
\begin{test}
\label{testa}
$n = 0$, $incx = 1$, $x = \{\}$ $\Rightarrow result
= 0.0$
\end{test}
\begin{test} $n = 1$, $incx = 0$, $x = \{\}$ $\Rightarrow result
\label{testb}
= 0.0$
\end{test}
We choose \fvarn{n} and \fvarn{incx} to be boundary values;
i.e., they are values closest to legal values.


$n=1$ could be treated as a special case although this is
very unlikely to provide any efficiency gain since it is not
a common practical occurrence. 
Treating this as a special case for real input data could cause
an inconsistency with the setting of the IEEE status flags;
for example, consider $x = \maxreal$ using the proposed 
hybrid method, then

\begin{enumerate}
\item treating this as a special case would give a result of
$\maxreal$ with the inexact flag not triggered,

\item\label{partb} not treating it as a special case
would generally generate the same result but the inexact flag 
would be set due to
the computation within Kahan's algorithm (two scalings,
a square and a square root) on a number which is not an exact
power of two,

\item $n=1$ is not treated as a special case for complex
input where it is equivalent to a real vector of
length two with $x=\{\maxreal, zero \}$ and this causes the
inexact flag to be set as in~\ref{partb} above.
\end{enumerate}

Finally, a vector of length $n$ where $x_i=\maxreal$ and $x_j=zero$
for $j \ne i$ will also cause the inexact flag to be set.

We therefore choose not to treat $n=1$ as a special case
(resulting in a generally irrelevant loss of accuracy
for real vectors of length one!). We do, however, include
a set of tests with $n=1$ as these provide

\begin{enumerate}
\item simple tests for our software, and
\item tests for an implementation that does treat $n=1$
as a special case. 
\end{enumerate}

We also include a test (Test~\ref{testn}) of the form
$\{0,0,\ldots,0,\maxreal,0,\ldots,0\}$ to check for
consistency (or to detect $n=1$ being treated as special).

\begin{test} $n=1$, $incx=1$, $x=\left \{3.0 \right \}$
\label{testc}
$\Rightarrow result = 3.0$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left \{-3.0 \right \}$
\label{testd}
$\Rightarrow result = 3.0$
\end{test}
The \fvarn{ieee\_inexact} status flag is not set in the next
four tests (\ref{teste}--\ref{testh}) as the exact result
can be represented; this will happen if $n=1$ is treated as
a special case or the method uses $x_1$ as a scale factor.
\begin{test}\label{teste}
$n=1$, $incx=1$, $x=\left\{ \maxreal \right\}$
$\Rightarrow result = \maxreal$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left\{ -\maxreal \right\}$
\label{testf}
$\Rightarrow result = \maxreal$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left\{ \Tiny \right\}$
\label{testg}
$\Rightarrow result = \Tiny$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left\{ -\Tiny \right\}$
\label{testh}
$\Rightarrow result = \Tiny$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left \{+Inf, \right \}$
\label{testi}
$\Rightarrow result = +Inf$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left\{ -Inf \right\}$
\label{testj}
$\Rightarrow result = +Inf$
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left\{ sNaN \right\}$
\label{testk}
$\Rightarrow result = qNaN$
(\fvarn{ieee\_invalid} set)
\end{test}
\begin{test} $n=1$, $incx=1$, $x=\left\{ qNaN \right\}$
\label{testl}
$\Rightarrow result = qNaN$
(\fvarn{ieee\_invalid} set)
\end{test}

Check that there are no problems if the algorithm scales and the 
input vector contains only zero elements:

\begin{test} $n=10$, $incx=1$, 
\label{testm}
$x=\left\{ 0, 0, \ldots, 0, 0 \right\}$
$\Rightarrow result = 0$
\end{test}

Test for vector of all zeros except for a
single element of value $\maxreal$:

\begin{test} $n=5$, $incx=1$, 
\label{testn}
$x=\left\{ 0, 0, \maxreal, 0, 0 \right\}$
$\Rightarrow result = \maxreal$
(the comments preceding Test~\ref{teste} also apply here)
\end{test}

For the two classes~\ref{test:nonfinite} and~\ref{test:nans} we
choose an input vector that contains infinite or
NaN values. For completeness we use both positive
and negative infinity along with signaling and quiet NaN
values:
\begin{test} $n=3$, $incx=1$, $x=\left \{ 2.0, -1.0, +Inf
\label{testo}
\right \}$
$\Rightarrow result = +Inf$
\end{test}
\begin{test} $n=3$, $incx=1$, $x=\left\{ 2.0, -Inf, -1.0
\label{testp}
\right\}$
$\Rightarrow result = +Inf$
\end{test}
\begin{test} $n=3$, $incx=1$, $x=\left\{ 2.0, -Inf, sNaN
\label{testq}
\right\}$
$\Rightarrow result = +Inf$
\end{test}
\begin{test} $n=3$, $incx=1$, $x=\left\{ 2.0, sNaN, -1.0
\label{testr}
\right\}$
$\Rightarrow result = qNaN$
(\fvarn{ieee\_invalid} set)
\end{test}
\begin{test} $n=3$, $incx=1$, $x=\left\{ qNaN, 2.0, -1.0
\label{tests}
\right\}$
$\Rightarrow result = qNaN$
(\fvarn{ieee\_invalid} set)
\end{test}
\begin{test} $n=4$, $incx=1$, $x=\left\{ qNaN, sNaN,
\label{testt}
-Inf, sNaN \} \right\}$
$\Rightarrow result = +Inf$
\end{test}

For class~\ref{test:nonrep} we need to consider a vector of
size at least two since, for $n=1$, if the element is a 
representable
floating-point value, then we can represent the value of the $l_2$
norm as it is just the absolute value of the input and all valid
negative IEEE values have a valid positive value of the same
magnitude. Hence, an obvious test would be: 
\begin{test} $n=2$, $incx=1$, $x=\left\{\maxreal,
\label{testu}
\maxreal\right\}$
$\Rightarrow result = +Inf$ (\fvarn{ieee\_overflow} and
\fvarn{ieee\_inexact} set)
\end{test}

The next two data sets use boundary values to test that the
routine returns a finite value/overflows at the expected
point. Use $x=\left \{ \maxreal, x_2 \right \}$ and choose
$x_2$ to be the largest value which won't cause an overflow
and the smallest value which will using floating-point
arithmetic. We have

\begin{equation}
res = \sqrt{\maxreal^2 + (x_2)^2} = \maxreal \sqrt{\left (1 + \frac
{x_2^2}{\maxreal^2}\right )}
\end{equation}
If the software is performing to its specification, we should
obtain an infinite result only if
$(1+x_2^2/\maxreal^2)>1$ otherwise a value of
$\maxreal$ should be returned. Thus an unrecoverable overflow
would occur if $x_2^2 \ge \maxreal^2 \macheps$ and
using $\maxreal = 2^{\Omega}$ to obtain
an exact power of two
gives $x_2 \ge 2^{\Omega+\halfeps}$.

Whence,
\begin{test} $n=2$, $incx=1$, $x=\left\{\maxreal, 
\label{testv}
2^{\Omega+\halfeps}\right \}$
$\Rightarrow result = +Inf$ 
(\fvarn{ieee\_overflow} and \fvarn{ieee\_inexact} set)
\end{test}
\begin{test} $n=2$, $incx=1$, $x=\left\{\maxreal, 
\label{testw}
2^{\Omega+\halfeps-1}\right \}$
$\Rightarrow result = \maxreal$ (\fvarn{ieee\_inexact} set)
\end{test}

We should also check that we do not have any problems with
underflows:

\begin{test} $n=4$, $incx=1$, $x=\left\{\Tiny, \Tiny, \Tiny,
\label{testx}
\Tiny\right\}$
$\Rightarrow result = 2*\Tiny$
\end{test}

Next we consider inputs for which the sum of squares of the
elements is exactly representable as an IEEE floating point
value.

The first test may cause the inexact flag to be set if non-powers of two
are used for scaling, however, the exact result is
representable. The second test should set the inexact flag
and should agree with the value provided by the Fortran square
root intrinsic.

\begin{test} $n=3$, $incx=1$, $x=\left\{9.0, 18.0, 38.0 \right\}$
\label{testy}
$\Rightarrow result = 43.0$
\end{test}
\begin{test} $n=3$, $incx=1$, $x=\left\{10.0, 11.0, 12.0\right\}$
\label{testz}
$\Rightarrow result = \sqrt{365}$ (\fvarn{ieee\_inexact} set)
\end{test}

It would also be useful to have an example for a larger
value of $n$.  \citet{Hirschhorn:2011:WSC} provides the
following

\begin{equation}
\label{eq:pythag}
\sum_{i=0}^{k-1} (a+i)^2 = \sum_{i=0}^{k-1} r_i^2 = b^2
\end{equation}
where $k=v^2$, $a=(v^4-24v^2-25)/48$ and $v$
is any integer not divisible by 2 or 3.
The required sum is then $b^2$ where $b$ is given by
\begin{equation}
\label{sumRes}
b = (v^5+47v)/48
\end{equation}

As $v$ increases both the starting value, $a$, and the
number of terms also increase. We note that all the
values are integer and, hence, for smaller values of $v$
the sum of squares will be collected exactly.  We denote
the largest value of $v$ that allows an exact result to be
computed in the chosen precision by $v_1$; the next valid
value of $v=v_2$ should generate an inexact result.  For IEEE
single and double precision the values for $(v_1, v_2)$
are $(11,13)$ and $(85, 89)$ respectively. The subroutine
\texttt{getVals} in the module \texttt{sdTestMod} attempts
to compute these values automatically.  Hence

\begin{test} $n = v_1^2$, $incx = 1$, $x=\left\{
\label{testA}
r_0, r_1, \ldots , r_{n-1} \right \}$ from
equation~(\ref{eq:pythag})
$\Rightarrow result = 3366.0$ and $92438690.0$ for IEEE single and double precision respectively.
\end{test}
would provide an extended test. We should also add

\begin{test} $n = v_2^2$, $incx = 1$, $x=\left\{
\label{testB}
r_0, r_1, \ldots , r_{n-1} \right \}$ from
equation~(\ref{eq:pythag})
$\Rightarrow result = 7748.0$ and $116334659.0$ for IEEE single and double precision respectively.
(\fvarn{ieee\_inexact} set)
\end{test}
to ensure that the IEEE flags are set correctly on exit.

The following six tests would all cause the simple loop to
overflow or underflow. The first two should generate the exact
result whether or not the scaling factors used are a power of
two; the second pair also have an $l_2$ norm which is exactly
representable but using scaling factors that are not powers
of two will probably lead to the inexact flag signalling.
\begin{test}
\label{testC}
$n = 4$, $incx=1$,
$x=\left\{p,p,p,p\right\}$
$\Rightarrow result = 2*p$
\end{test}
\begin{test}
\label{testD}
$n = 4$, $incx=1$,
$x=\left\{q,q,q,q\right\}$
$\Rightarrow result = 2*q$
\end{test}
\begin{test}
\label{testE}
$n = 4$, $incx=1$,
$x=\left\{9*p,11*p,13*p,23*p\right\}$
$\Rightarrow result = 30*p$
\end{test}
\begin{test}
\label{testF}
$n = 4$, $incx=1$,
$x=\left\{9*q,11*q,13*q,23*q\right\}$
$\Rightarrow result = 30*q$
\end{test}
The last pair of tests should both generate results with the
inexact flag set due to the square root operation
\begin{test}
\label{testG}
$n = 5$, $incx=1$,
$x=\left\{p,p,p,p,p/4\right\}$
$\Rightarrow result = \sqrt{65}p/4$
(\fvarn{ieee\_inexact} set)
\end{test}
\begin{test}
\label{testH}
$n = 5$, $incx=1$,
$x=\left\{q,q,q,q,2*q\right\}$
$\Rightarrow result = 2\sqrt{2}q$
(\fvarn{ieee\_inexact} set)
\end{test}
where $p=2^{\Omega/2-1}$ and $q=2^{-\Omega/2}$.

In the next test the squaring would result in an
unnormalized number, if they are available, or zero,
if they aren't. Either way an underflow would occur with
the simple loop. Kahan's algorithm will use scaling to
generate an inexact, but accurate, result.

\begin{test}
\label{testI}
$n = 4$, $incx=1$,
$x=\left\{p,p,p,p\right\}$
$\Rightarrow result = 2*p$
where $p = r^{-\Omega/2}$ and $r$ is the nearest floating point number less than one.
(\fvarn{ieee\_inexact} set)
\end{test}

The following two tests are designed to show up scaling
problems within the elements of the vector. The value of
$p$ is chosen so that $1+p^2$ returns 1 under the floating
point arithmetic being used. The vector is then made large
enough so that $1+\sum p^2$ would be representable. Using a
naive approach the first test will return 1 while a similar
approach to the second should provide an accurate result.

\begin{test}
\label{testJ}
$n = 2^{nval}+1$, $incx=1$,
$x=\left\{1,p,p,p,\ldots, p\right\}$
$\Rightarrow result = \sqrt{1+p^2*2^{nval}}$
(\fvarn{ieee\_inexact} set)
\end{test}

\begin{test}
\label{testK}
$n = 2^{nval}+1$, $incx=1$,
$x=\left\{p,p,p,\ldots, p, 1\right\}$
$\Rightarrow result = \sqrt{1+p^2*2^{nval}}$
(\fvarn{ieee\_inexact} set)
\end{test}
where $p=2^{\halfeps-2}$ and $nval=10$.

The next tests check that the implementation operates
correctly when $incx \ne 1$ on a series of simple examples.

\begin{test}
\label{testL}
$n = 3$, $incx=2$,
$x=\left\{ 2.0, 5.0, 3.0, 9.0, 6.0\right\}$
$\Rightarrow result = 7.0$
\end{test}

\begin{test}
\label{testM}
$n = 2$, $incx=3$,
$x=\left\{2.0, 3.0, 4.0, 5.0\right\}$
$\Rightarrow result =\sqrt{29} $
(\fvarn{ieee\_inexact} set)
\end{test}

\begin{test}
\label{testN}
$n = 3$, $incx=2$,
$x=\left\{2.0, +Inf, 3.0, qNaN, 6.0\right\}$
$\Rightarrow result = 7.0$
\end{test}

The next two tests are algorithm specific and attempt to check that the different accumulator elements are combined correctly; to be specific: medium and big; and small and medium.

\begin{test}
\label{testO}
$n = 2$, $incx=1$,
$x=\left\{2^{c-1}, 2^{c+1}\right\}$
$\Rightarrow result = \sqrt{17}\times2^{c-1}$
(\fvarn{ieee\_inexact} set)
\end{test}

\begin{test}
\label{testP}
$n = 2$, $incx=1$,
$x=\left\{2^{d-1}, 2^{d+1}\right\}$
$\Rightarrow result = \sqrt{17}\times2^{d-1}$
(\fvarn{ieee\_inexact} set)
\end{test}
\noindent
where $B=f_B2^c$ and $b=f_b2^d$ are defined in Blue's algorithm
and $c=EXPONENT(B)$, $d=EXPONENT(b)$, $f_B=FRACTION(B)$ and
$f_b=FRACTION(b)$ as provided by the Fortran 2008 elemental
functions with $f_b, f_B \in \left [ \frac{1}{2}, 2 \right )$
for IEEE floating point arithmetic.

The next six tests check that the IEEE status flags are
processed correctly. We have tested that the flags are set
correctly in the above tests when the input status flags are
all set to FALSE on entry. The following tests mostly check
that when the input flags have all been set to TRUE then the
flags are all still TRUE on exit whether the call to \nrm
would have signalled or not.

In these six tests the status flag settings are stored in a
logical array of length five in the order

$$
\left\{ ieee\_overflow, ieee\_divide\_by\_zero, ieee\_invalid,
ieee\_underflow, ieee\_inexact \right \}
$$
\noindent
We give the definition of the test problem and the state of
the input flags along with the expected result and exit settings
for the flags.

\begin{test}
\label{testQ}
$n = 2$, $incx=1$,
$x=\left\{5, 12\right\}$, $status=\left\{T,T,T,T,T\right\}$
$\Rightarrow result = 13$, $status=\left\{T,T,T,T,T\right\}$ 
\end{test}

\begin{test}
\label{testR}
$n = 2$, $incx=1$,
$x=\left\{7, 9\right\}$, $status=\left\{T,T,T,T,T\right\}$
$\Rightarrow result = \sqrt{130}$, $status=\left\{T,T,T,T,T\right\}$ 
\end{test}

\begin{test}
\label{testS}
$n = 2$, $incx=1$,
$x=\left\{7, 9\right\}$, $status=\left\{T,T,T,T,F\right\}$
$\Rightarrow result = \sqrt{130}$, $status=\left\{T,T,T,T,T\right\}$ 
\end{test}

\noindent
In the next three tests $p=2^{\Omega/2-1}$

\begin{test}
\label{testT}
$n = 4$, $incx=1$,
$x=\left\{p, p, p, p\right\}$, $status=\left\{T,T,T,T,T\right\}$
$\Rightarrow result = 2*p$, $status=\left\{T,T,T,T,T\right\}$ 
\end{test}

\begin{test}
\label{testU}
$n = 5$, $incx=1$,
$x=\left\{p, p, p, p, p/4\right\}$, 
$status=\left\{T,T,T,T,T\right\}$
$\Rightarrow result = \sqrt{65}p/4$, 
$status=\left\{T,T,T,T,T\right\}$ 
\end{test}

\begin{test}
\label{testV}
$n = 5$, $incx=1$,
$x=\left\{p, p, p, p, p/4\right\}$, 
$status=\left\{T,T,T,T,F\right\}$
$\Rightarrow result = \sqrt{65}p/4$, 
$status=\left\{T,T,T,T,T\right\}$ 
\end{test}

Finally, these tests require the floating-point system to
provide gradual underflow and they check that underflows
are not triggered for input vectors containing elements
in the gradual underflow range.

\begin{test}
\label{testW}
$n = 4$, $incx=1$,
$x=\left\{p,p,p,p\right\}$
$\Rightarrow result = 2*p$
where $p = \Tiny/4$
\end{test}

\begin{test}
\label{testX}
$n = 4$, $incx=1$,
$x=\left\{p,p,p,p\right\}$
$\Rightarrow result = 2*p$
where $p$ is the smallest representable real value $= 2^{\minExponent-\digits}$
\end{test}


\begin{test}
\label{testY}
$n = 4$, $incx=1$,
$x=\left\{p,p,p,p\right\}$
$\Rightarrow result = 2*p$
where $p$ is the next representable real value $> \Tiny$
\end{test}

\input{BigTab} 

\subsection{Output of IEEE arithmetic status flags}%
\label{subsec:ieeeflags}

The package checks and reports on the final settings of the
five IEEE floating-point status flags that can be set during
the execution of the test suite:
\begin{enumerate}
\item overflow
\item divide by zero
\item invalid
\item underflow
\item inexact
\end{enumerate}
and outputs the values as a string of five characters separated by spaces. The order of the characters in the output is the same as the order of the flags given above.

Each flag may take the value
\begin{description}
\item[T:] status flag has signalled during the computation of the norm,
\item[F:] status flag has not  signalled,
\item[X:] the status flag is not supported by the compiler/platform combination being used.
\end{description}

For example, the string \textit{T\_F\_X\_T\_F} would signify that 
\begin{itemize}
\item the overflow and underflow flags signalled,
\item the divide by zero and inexact flags did not signal, and
\item the invalid flag was reported as not being supported.
\end{itemize}

If any of the characters used in the string offends the user they may be changed by altering the values of \fvarn{chTrue} (flag signalling), \fvarn{chFalse} (flag not signalling) and \fvarn{chUnsupported} (flag not supported) in the module variables section of the module \fvarn{chTests}.

A discussion of the testing and reporting of the inexact flag may be found at the start of this section.

\subsection{Controlling computation and output of the test suite}%
\label{subsec:testcontrol}
The user may control
\begin{enumerate}
\item the amount of detail output when running the test suite, and
\item what results are compared and reported.
\end{enumerate}
Different controls may be applied independently to the
single and double precision versions of the test suite.

The level of output obtained may be altered using
the routine \fvarn{setOutputLevel} from the module
\fvarn{checkTests}. This routine takes a single integer
argument \fvarn{level}; for example

\begin{lstlisting}
       CALL setOutputLevel(2)
\end{lstlisting}
sets the output level to 2. The level set remains in force
throughout the execution of the test suite unless it is
altered by a subsequent call. Thus we may obtain different
levels of output for the different precisions using

\begin{lstlisting}
! Set output level for double precision tests
       CALL setOutputLevel(2)
! ... double precision test calls ...

       CALL setOutputLevel(1)
! ... single precision test calls ...
\end{lstlisting}

The default level is \fvarn{level = 3} and this level
will be used if no calls to \fvarn{setOutputLevel} have
been made prior to the test suite being executed. If the
value provided to \fvarn{setOutputLevel} is outside of the
range $[1, 3]$ then the value 3 is used. The current
value of \fvarn{level} may be obtained using the function
\fvarn{getOutputLevel} which returns the value as an integer:

\begin{lstlisting}
       currentLevel = getOutputLevel()
\end{lstlisting}

The different levels available are
\begin{itemize}
\item \fvarn{level = 1}: full output of all test results.

Results are printed for each individual test and for both
real and complex input vectors whether or not the test
fails. The output includes

\begin{itemize}
\item the title of the test,
\item the type of the input vector (\fvarn{REAL} or \fvarn{COMPLEX}),
\item the computed $l_2$-norm,
\item the returned IEEE arithmetic status flags,
\item a count of the number of failures reported.
\end{itemize}
For both the results and the status flags the program also
outputs either \textit{Correct} or \textit{***Incorrect***}
tags after comparing the computed values with the expected
values. If any of the computed results do not agree, the
expected results are also output along with a `banner'
output line. For example:

\begin{verbatim}
Test  22: Boundary value causing overflow
Using REAL input vector
Result returned =       +Infinity  Correct  
IEEE status flags returned:     T F F F T **Incorrect**
                  expected:     F F F F T
********* TEST FAILED **************
\end{verbatim}

In the case where the computational result differs from the expected, a line giving the expected value and the relative difference 
\begin{equation}
\left | \frac{expected - computed}{expected} \right |
\nonumber
\end{equation}
is output unless the expected output is zero when the value
\begin{equation}
\left | computed \right |
\nonumber
\end{equation}
is used.

\item \fvarn{level = 2}: compressed output of all test results.

As with \fvarn{level = 1} above this level produces output
for all the test cases but uses a more compact form from the
detailed level. If the computed results agree with the expected
then a single line is output and tagged \textit{Correct}. If any
differences occur then, in the case of the computed result,
the error in the returned value is given (as detailed in
\fvarn{level = 1} above). When the IEEE status flags do not
agree then the expected flags are output directly underneath
the returned values. In both cases the incorrect test output
is highlighted by using the tag \textit{**Incorrect**}. A count of the number of failures is also reported.

For example:
\begin{verbatim}
  22 R         +Infinity       NONE         T F F F T **Incorrect**
                                            F F F F T
\end{verbatim}

\item \fvarn{level = 3}: compressed output of unexpected test results.

The output is the same as \fvarn{level = 2} above but only
occurs if differences are detected between the computed and
expected results. A count of the number of failures is also reported. 

\end{itemize}

\subsection{Controlling the scope of the testing}%
\label{subsec:scope}

By default the test driver will check both the return values of the $l_2$ norm and the settings of the IEEE status flags. It is also possible just to check the returned values and ignore the status flags in both the test and the report.

The routine \fvarn{setJustValues} from the module \fvarn{checkTests} has the form
\begin{lstlisting}
       CALL setJustValues(value)
\end{lstlisting}
where \fvarn{value} is a simple logical variable. If \fvarn{value} is set to \fvarn{.TRUE.} then only the computed values are compared and reported. If \fvarn{value} is set to \fvarn{.FALSE.} (the default) then both computed values and the IEEE status flags are compared and reported.

The current setting may be retrieved using
\begin{lstlisting}
       value = getJustValues()
\end{lstlisting}

\subsection{The datatype \textit{testInfoType}}\label{subsec:type}

The data type \textit{testInfoType} is used to store the definition
of a test, the expected computed value and returned IEEE
status flags, the returned computed value and the returned IEEE
status flags, and a title/short description of the test which
is available for output. It is defined as follows:

\begin{lstlisting}
      TYPE testInfoType
        INTEGER :: testNo
        CHARACTER(LEN=:), ALLOCATABLE :: testDescript
        REAL(wp), POINTER :: inVector(:)
        INTEGER :: nVal, incxVal
        LOGICAL :: initialStatus(5)
        REAL (wp) :: expectedNorm
        LOGICAL :: expectedStatus(5)
        REAL (wp) :: returnedNorm
        LOGICAL :: returnedStatus(5)
      END TYPE testInfoType
\end{lstlisting}

A short description of each component follows:

\begin{description}
\item[\fvarn{testNo}] is the test number associated with the test. This is used
to identify the particular test in the output. Details of how to add extra tests to the suite given in Section~\ref{sec:addnewtest}.

\item[\fvarn{testDescript}] contains a short description of the test and is
output for some levels of output.
\end{description}

\noindent
The next three components define the data to be used in the test.
The call to an \nrm routine is of the form
\begin{lstlisting}
       returnedNorm = *nrm2(nVal, inVector, incxVal)
\end{lstlisting}

\begin{description}
\item[\fvarn{inVector}] is the data for the real array -- for details of how the
equivalent complex vector is generated see the start of this section.

\item[\fvarn{nVal}] is the number of elements to be used in computing the norm,

\item[\fvarn{incxVal}] is the gap to be used between elements; elements used are
$(1, incxVal+1, 2*incxVal+1, \ldots, (nVal-1)*incxVal + 1)$,      

\item[\fvarn{initialStatus}] defines the IEEE status flags on entry to the \nrm
routine. These may be used for two purposes:

\begin{enumerate}
\item set to all FALSE to test that the \nrm implementation
under test only signals the correct flags,

\item The flags can also be set to test that the correct union
of flags is returned after executing the \nrm routine,
i.e., the call to the \nrm routine does not adversely
affect any existing settings.
\end{enumerate}

\item[\fvarn{expectedNorm}] defines the expected return value from the call to
the \nrm routine. This will be the same value for the
real and complex calls for a given precision. This should
be the correctly rounded representation of the exact value.

\item[\fvarn{expectedStatus}] defines the expected values of the IEEE status flags
on exit from the \nrm routine. This should take account of
the initial setting.

\item[\fvarn{returnedNorm}] is the computed value returned by the call
to the \nrm routine.

\item[\fvarn{returnedStatus}] is the vector of IEEE status flags following the
call to the \nrm routine.
\end{description}


\subsection{Adding a New Test Case}\label{sec:addnewtest}
The code defining the individual test cases within the test suite is linear -- each test being executed in its order of appearance within the file \filename{sdTestMod.inc}.

The test number assigned to the test depends on its order within the sequence. Thus adding a new test in the middle of the existing suite will move all the following test numbers out of alignment 
with the descriptions in Section~\ref{sec:Testing}. It is, therefore, recommended that additional tests are added after the comment
\begin{lstlisting}
! Any new tests should be added here in the first instance to
! prevent misaligning test numbers with the descriptions given
! in the user manual.
\end{lstlisting}
just before the definition of \fvarn{SUBROUTINE getVals} in the file \filename{sdTestMod.inc}.

\subsubsection{Format of the Test Definition}\label{subsec:newtest}

In the following, names like \fvarn{nval},
\fvarn{incxVal}, etc are shortened forms of the components
of the \fvarn{testInfoType} defined via the \fvarn{ASSOCIATE}
statement:

\begin{lstlisting}
        ASSOCIATE(testNo=>testInfo%testNo, &
                  expStat=>testInfo%expectedStatus, &
                  expRes =>testInfo%expectedNorm, &
                  nVal => testInfo%nVal, &
                  incxVal => testInfo%incxVal, &
                  initStat => testInfo%initialStatus)
\end{lstlisting}
The components \fvarn{inVector} and \fvarn{testdescript}
are dynamic and are quoted in full.

We use an existing test as an example of how a new test
should be defined:

\lstset{numbers=left, numberstyle=\tiny}
\begin{lstlisting}
! Vector of length = 4 -- incx = 3, n=2 -- inexact result
        ALLOCATE (x(4))
        testInfo%inVector=>x
        x = [two, three, four, five]
	nVal = 2
	incxVal = 3
	initStat = allFalse
        expStat =  [.FALSE.,.FALSE.,.FALSE.,.FALSE.,.TRUE.]
        expRes = SQRT(twentynine)
        testNo = testNo + 1
        testInfo%testdescript = 'incx = 3, inexact result'
        CALL indivtest(testInfo)
        DEALLOCATE (x)
\end{lstlisting}
\lstset{numbers=none, frame=single}

\begin{description}
\item{Line 1:} provides a comment describing the test that
follows,

\item{Line 2:} \fvarn{ALLOCATE} just the space needed to
contain the input vector, \fvarn{x},

\item{Line 3:} set the component \fvarn{inVector} of the
variable \fvarn{testInfo} of type \fvarn{testInfoType} to
point to the vector \fvarn{x}. \fvarn{testInfo} contains
a number of components that constitute all the required
data to run and check an individual test.

\item{Line 4:} define the data in the \fvarn{x} vector. This
may be achieved using a block of code or, for shorter examples,
an array initialization,

\item{Line 5:} set the number of values to be used. This
should either be the number of elements in the input vector,
\fvarn{x}, i.e., \fvarn{SIZE(x)} when \fvarn{incxVal} is one or
the number of elements that should be used if \fvarn{incxVal}
is greater than one; i.e., \fvarn{x(j)} for \fvarn{j} $= 1,
1+incxVal, \ldots, 1+incxVal \times (nVal - 1)$.

\item{Line 6:} set the increment to be used when accessing
the \fvarn{nVal} elements within the vector \fvarn{x}; i.e.,
\fvarn{x(j)} for \fvarn{j} $= 1, 1+incxVal, \ldots, 1+incxVal
\times (nVal - 1)$.

\item{Line 7:} define the initial settings for the IEEE status
flags on entry to the \nrm routine. Usually these will be
set to all \fvarn{.FALSE.} (the \fvarn{LOGICAL, PARAMETER}
array \fvarn{allFalse} may be used to do this) so that only
the flags set to signalling by the routine are visible. This
array may also be used to test that any signalling flags are
not incorrectly reset when the \nrm routines are called.

\noindent
The order of the flags is:
\begin{equation*}
\left [ overflow, divide-by-zero, invalid, underflow, inexact \right ]
\end{equation*}

\item{Line 8:} define the expected settings of the IEEE floating
point status flags on exit from the calls to each of the \nrm
routines (it is assumed that the same flags will be returned
for all four of the routines).

\item{Line 9:} set the expected result. This should be the
result obtained when taking into account the use of floating
point arithmetic.

\item{Line 10:} increment the test number, \fvarn{testNo}.

\item{Line 11:} set a short descriptive title for the
test. This is only output when detailed output is requested
(see Section~\ref{subsec:testcontrol}).

\item{Line 12:} makes the call to execute the individual test
using \fvarn{indivTest}. There is one mandatory argument,
\fvarn{testInfo}, which is a structure, of type
\fvarn{testInfoType} (see Section~\ref{subsec:type} for details)
holding all the information required to run the test and
compare the computed result against the expected.

\item{Line 13:} \textit{DEALLOCATE} the input data vector, \fvarn{x}.

\end{description}
