Dear Electricity Authority,
Via the centralised dataset, data on the workings of the electricity system have been publicised. However, for the 26'th of March 2011 an argument over an "Undesirable Trading Situation" began and dragged on. One result was that half-hourly data for the "Final Prices" series were suppressed for the 26'th.
This is to request a copy of the data that were originally supplied for that date, not the values that were (years later) publicised after the litigation was resolved.

The Final Prices data collection comprises some two hundred and fifty-four data series (on the 25'th and the 27'th of March 2011) and so two hundred and fifty-four data series are anticipated. Further, each half-hourly final price value has as well a timestamp, giving the date and time of the calculation of that value. For instance,
ABY0111,25/03/2011,48,F,46.57,26/03/2011 07:43:36
represents a final price datum for the date series encoded "ABY0111" for the 25'th of March 2011; it is the last half-hourly value for that day, thus half-hour 48; the "F" signifies a "final" price; and 46.57 is the datum. This is followed by the timestamp for the calculation, which is after the indicated time of the datum. Similarly,
ABY0111,27/03/2011,1,F,23.44,28/03/2011 09:45:59
is the first value for that data series after the gap.
And there are two hundred and fifty-four such data series, with code names from ABY0111 to WWD1103, as follow: ABY0111, ADD0111, ADD0661, ALB0331, ALB1101, APS0111, ARA2201, ARG1101, ARI1101, ASB0331, ASB0661, ASY0111, ATI2201, ATU1101, AVI2201, BAL0331, BDE0111, BEN0162, BEN0163, BEN2201, BEN2202, BLN0331, BOB0331, BOB1101, BPD1101, BPE0331, BPE0551, BPE2201, BPT1101, BRB0331, BRK0331, BRY0111, BRY0661, BWK1101, CBG0111, CLH0111, CML0331, COB0661, COL0111, COL0661, CPK0111, CPK0331, CST0331, CUL0331, CYD0331, CYD2201, DAR0111, DOB0331, DOB0661, DVK0111, EDG0331, EDN0331, FHL0331, FKN0331, GFD0331, GIS0501, GLN0331, GLN0332, GOR0331, GYM0661, GYT0331, HAM0111, HAM0331, HAM0551, HAM2201, HAY0111, HAY0331, HAY1101, HAY2201, HEN0331, HEN2201, HEP0331, HIN0331, HKK0661, HLY0331, HLY2201, HOR0331, HOR0661, HTI0331, HUI0331, HWA0331, HWA0332, HWA1101, HWA1102, HWB0331, HWB0332, HWB2201, INV0331, INV2201, INV2202, ISL0331, ISL0661, ISL2201, KAI0111, KAW0111, KAW0112, KAW0113, KAW1101, KAW2201, KEN0331, KIK0111, KIN0111, KIN0112, KIN0113, KIN0331, KKA0331, KMO0331, KOE0331, KPA1101, KPO1101, KPU0661, KTA0331, KUM0661, KWA0111, LFD1101, LFD1102, LTN0331, MAN2201, MAT1101, MCH0111, MDN1101, MDN2201, MER0331, MGM0331, MHO0331, MLG0111, MLG0331, MLN0661, MLN0664, MNG0331, MNG1101, MNI0111, MOT0111, MPE0331, MPI0661, MST0331, MTI2201, MTM0331, MTN0331, MTO0331, MTR0331, NAP2201, NMA0331, NPK0331, NPL0331, NPL1101, NPL2201, NSY0331, OAM0331, OHA2201, OHB2201, OHC2201, OHK2201, OKI0111, OKI2201, OKN0111, ONG0331, OPK0331, ORO1101, ORO1102, OTA0221, OTA1101, OTA1102, OTA2201, OTA2202, OTI0111, OWH0111, PAK0331, PAL0331, PAP0111, PAP0661, PEN0221, PEN0331, PEN1101, PNI0331, PPI2201, PRM0331, RDF0331, RDF2201, RFN1101, RFN1102, ROS0221, ROS1101, ROT0111, ROT0331, ROT1101, ROX1101, ROX2201, RPO2201, SBK0331, SDN0331, SFD0331, SFD2201, SPN0331, SPN0661, STK0331, STK2201, STU0111, SVL0331, SWN2201, TAK0331, TGA0111, TGA0331, TIM0111, TKA0111, TKA0331, TKB2201, TKH0111, TKR0331, TKU0331, TKU2201, TMI0331, TMK0331, TMN0551, TMU0111, TMU1101, TNG0111, TNG0551, TRK0111, TRK2201, TUI0111, TUI1101, TWC2201, TWH0331, TWI2201, TWZ0331, UHT0331, WAI0111, WDV0111, WDV1101, WEL0331, WGN0331, WHI0111, WHI2201, WHU0331, WIL0331, WIR0331, WKM2201, WKO0331, WPA2201, WPR0331, WPR0661, WPT0111, WPW0331, WRA0111, WRK0331, WRK2201, WTK0111, WTK0331, WTK2201, WTU0331, WVY0111, WWD1102, WWD1103 - though in principle another name might appear - just for one day.

Thus, the request is for the suppressed final price data, the values with their timestamps, as they has been routinely calculated by the standard process as used in March 2011.

Yours faithfully,
R.N. McLean

Dear Christina Hammond,
Thank you for the response signed by Carl Hansen. Some technical issues arose, delaying my response. Similar technical details have affected the layout and typeface usage of my reply.

Dear Mr. Hansen,

The FYI system advised me of the receipt of your response, which was presented as a .pdf file bearing a smudged and blurred image of your letter, plus a .html option whereby it appears more clearly but rotated ninety degrees clockwise. After some enquiries with the FYI operators, I have received a copy of your original file which shows a rather better image. The FYI procedure is to reprocess such files to hide the likes of e-mail addresses from easy access by intending spammers who are “scraping” e-mail addresses from whatever computer-readable file that may be found. This reprocessing was triggered by the presence of plain text in the .pdf file along with the image. And their system does not cope well with .pdf files containing a “rotate” instruction.
When your letter (with its signature) was scanned to produce a .pdf file for electronic transmission, the printer/photocopier/scanner also initiated an attempt at optical character recognition in order to associate searchable text with the components of the image in the resulting .pdf file. Despite the boasts of salespersons, OCR is not reliable, and so the text contains mistakes, especially in the attempted recovery of web page links. In the first link, both occurrences of the letter “g” became the letter “q” and as well the line break within the url text was not recognised, resulting in a truncated url. Thus, although it was shown as available for rodent poke, the thrust did not arrive. Similar problems arise with the other links, especially those involving the use of the underscore character.
This is a second problem. It is routine in documents to have url texts marked by Cerenkov blue colouring, adding a touch of drama to an otherwise bland narrative, but it is also usual to have such links underlined. This of course spoils the visibility of the down strokes of those glyphs such as “g” and also makes it nearly impossible to determine whether the text is for example “Final pricing” or “Final_pricing” - though curiously enough it is the more blurred image that shows more of a hint of the underline character being present as well as the underlining. Hyphens are not affected.
The Open Office document editing system has an option “export as PDF” and for those using Microsoft Word there seem to be similar provisions; also it is possible to install a pseudo-printer whose output is a .pdf file. Such paper-equivalent output presumably would not include details from within a document file such as its revision history and annotations, etc. Given a document containing correct url texts, the resulting .pdf file would contain the correct text of the document with the urls correctly formed and unaffected by line splits. The recipient of such a .pdf file could then use its offered links without difficulty. Though I’m not sure how you could affix your signature. Alternatively, you could produce an image-only .pdf file: no searchable texts, no links, and no problem.
Enough of the syntax, now for the semantics.

After repairing your letter’s nominated link to obtain http://www.ea.govt.nz/code-and-complianc... I started reading...
Some confusion was produced by the likes of a link presented as UTS 26 March 2011 – Contact Energy request that resolved to http://www.ea.govt.nz/dmsdocument/11064 and immediately below that link in smaller text Contact-Energy-request-UTS-26-March-2011.pdf (PDF, 443KB) that resolved to http://www.ea.govt.nz/dmsdocument/11064 – the same document, when, given the different texts, two different documents might be expected. This occurred in other places as well (for instance, the “Correspondence” page) but because the same file name was used, I was able to avoid having to read those twice to discover their equivalence.
In reading UTS-Committee-Final-Decision-on-UTS-and-Actions.pdf, on page 15, clause (f) I was puzzled by the usage of [ ] in the style of a name as in “In addition, [ ] advised that...” - possibly this was a placeholder that had not been filled in. Likewise in clause 125 on page 36, clause 147 on page 39; enough. There was similar usage on page 71 et seq. apparently referring to money amounts. Is this perhaps a draft of the final decision? If not, why leave out the details?
Anyway, in all the discussion it soon became clear that the originally-calculated final price values were termed “interim” and that nowhere were they given in full, obviously because all parties had access to them via the WITS service operated by the NZX as a part of their ordinary buy/sell activities. By contrast, a pair of data files (spreadsheet style) were offered, containing some proposed replacement values.
So the original prices did exist, and were available, but were not included in the data files published by the Electricity authority in its twice-yearly comprehensive dataset. Even though the data log shows that the data file for March 2011 was first supplied with the complete set of data, then resupplied with the data for the twenty-sixth removed. Thus, I continue to say that those data were “suppressed” by the Electricity authority - as far as the recipients of its published data sets were concerned. However, it is possible that the data for the twenty-sixth as initially supplied, were not given the code “F” because the disputes could have become recognised as sufficiently loud before the monthly file was prepared from the at-the-time calculations and the initial routine derivation’s “F” notation might have been revised. If so, it could be argued that the final prices were not suppressed because they did not exist.
Thus, I remain interested in the prices as they had first been determined by the routine operation of the process before any dispute arose. In this regard, the prices offered via http://www.emi.ea.govt.nz/r/c5hhn and http://www.emi.ea.govt.nz/Datasets/Whole... are not what I am looking for. Although they offer no “price run time” to make their provenance clear, they are surely the post-adjustment values and do not incorporate the startlingly high original values that initiated the dispute.

You reject my request on the basis that the desired data are already publicly available and (with the usual g/q problem) direct my attention to http://www.emi.ea.govt.nz/Datasets/Whole... in particular to three files whose name starts MSS_25. But there are twenty-eight such files in the 2011 file directory, not three. Fortunately, the link www.emi.ea.govt.nz/Forum/chat/spd-case-f... offers an interpretation of the file names, and curiously enough, Dr. Bishop has offered as an example exactly the files for the 26’ th March 2011, the web page having been last changed “three weeks ago”... It seems that afer the MSS_ comes the number of the day. This means that the files will be ordered so that for a given day, files for all the months having that day will be together. Further, because the day number does not have a leading zero for months up to October, the ordering will not have day two next to day one but to day ten. Given that further on in the file name is the month number, and it always two digits, what is the advantage of this ordering over putting the date parts together as YYYYMMDD? It certainly escapes me. And since the web page presents the files names in reverse order, finding files for a given date is even more difficult. But Dr. Bishop has found them and named them.
Curiously, his file name list contains an error. The files named by the web page appear as follows:
MSS_251112011031100351_0X.ZIP 19/02/2014 8:20:21 p.m. 6,288 KB
MSS_251112011031100322_0X.zip 31/01/2014 11:07:17 a.m. 8,267 KB
MSS_251112011031100213_0X.ZIP 19/02/2014 8:20:32 p.m. 6,293 KB
But his list has ZIP for all three.Your web page support system does not recognise a file type of .ZIP and so assigns a “no name” thumbnail, while it does recognise .zip and assigns an image of a manila folder with a “zip” across it.
Moving on to the content, these three files expand to give eight hundred and fifty-eight files comprising 277MB of storage. The standard final price data files take around 17MB for a whole month’s worth of data in 2011 so a single day’s worth in that format should occupy around half a MB, a factor of five hundred less. The file collection comprises over two dozen different types of files, and there is no obvious interpretation of them. But “the information requested is publicly available” so, given that code ABY0111 is the first named, perhaps something might be found. And something is. Guesswork suggests that the 18:00 before the ABY is a time, especially since it follows a date (in yet another format), and it transpires that three of the eight hundred and fifty-eight files each offer one line:
D,SOLUTION,PNODE,1.0,26-MAR-2011 18:00,ABY0111,X,0, -3.3890, 0.0000, 0.0000, 19.0824, 1.0018,NULL, 0.0000, 0.0000, 0.0000
D,SOLUTION,PNODE,1.0,26-MAR-2011 18:00,ABY0111,X,0, -3.3890, 0.0000, 0.0000, 19.0824, 1.0018,NULL, 0.0000, 0.0000, 0.0000
D,SOLUTION,PNODE,1.0,26-MAR-2011 18:00,ABY0111,X,0, -3.3890, 0.0000, 0.0000, 19.0824, 1.0018,NULL, 0.0000, 0.0000, 0.0000
each line being the same, and a further three offer
D,MSSDATA,PNODE,1.0,26-MAR-2011 18:00,ABY0111 ,L ABY 11 11KVLOAD ,X,1,ABY,11,11KVLOAD,LD,SI
D,MSSDATA,PNODE,1.0,26-MAR-2011 18:00,ABY0111 ,L ABY 11 11KVLOAD ,X,1,ABY,11,11KVLOAD,LD,SI
D,MSSDATA,PNODE,1.0,26-MAR-2011 18:00,ABY0111 ,L ABY 11 11KVLOAD ,X,1,ABY,11,11KVLOAD,LD,SI
also the same.
Presumably for other code names and times there would be differing results, and then arises the question as to which is to be selected and why. All this is very impressive. However, there are many numbers on each line, and none of them offer a value with the two decimal digits as are given in the monthly final price files, nor is there any obvious indication to be seen of a “price run time” either.
In the absence of any description of the format of these files, finding the desired data, even if present, will not be possible, and guesswork is insufficient because many questions will be unresolvable. For example, it is already explained that the file name’s day number is the day in UTC time, not New Zealand time, thus data for the 26’th are to be found in files with day 25 in their file name. Leaving aside what happens to the files for the first and last days of the year, it is evident that day 26 in NZ spans day 25 and 26 in UTC time, so perhaps there are two sets of such files needed? Further, during daylight saving time, NZ time is thirteen hours ahead, not twelve. Given that UTC times are used for the day number, what times are used for the date and time within the data files: is 18:00 a UTC time, the time on the NZ civil clock, or what? And what happens during the day of twenty-five hours? Not that this matters here. So, does the data file for day 25 UTC contain all of day 26 NZ time? If it does, why mix UTC and NZ time?
There are also questions of provenance. In the schedule of file names above, the three files offered by your system are shown with timestamps in 2014, not 2013 when the revised values were settled upon, nor yet 2011 when the values were first calculated. Further, the peculiar appearance of .zip instead of .ZIP for just these files suggests that they may not be the result of the original routine process with the original files routinely received and stored.

In short, if the data I have requested are in these publicly-available files as you say, I reply that they are unidentifiable as such without the knowledge to interpret the data formats correctly, and in the absence of a format description, the requested data are not in practice publicly available.
A copy of the data file for March 2011 (as supplied but then discarded in favour of the file omitting the 26’th) would be ideal; failing that, its reconstitution will require some effort, routinely performed by the supplier of the original data files every month. Otherwise, some greater effort will be required, guided by the description of the format of these data files. Presumably there is one?

Yours faithfully,
Nicky McLean.

Dear Christina Hammond,
Thank you for the reply containing the originally-computed "final" price data from the 26'th of March 2011. As before the "govt" became "qovt" but the data in the following table could be extracted via copying (using the free "Nitro" PDF reader) and seem to be clear. Certainly, there appear values in excess of $20,000 as had provoked all the trouble so these are indeed not the current data, as requested. Thank you for your trouble.
With regard to the "price run time" associated with every price value such as appear in the 2011 data files as also requested, I suppose the time 8:00am on 27 March 2011 that is associated with the data file you name could be used instead of 11:00am on 28 March 2011 (these appear in Dr. Bishop's "chat" entry http://www.emi.ea.govt.nz/Forum/chat/spd... and which you refer to again) that describes the file naming arrangements, but not their downloading nor their content. There is no particular difficulty in downloading these SPD case files, it lies in the interpretation of their (voluminous) content, and without any description of the internal format of the files to peruse, guesswork is unpromising. If there is such a description, I imagine that others would be interested in it also.

However, the grist are now to hand. Thank you again.

Yours sincerely,

Nicky McLean

Dear Christina Hammond,
I had thought that the Official Information Act request "Supply the original 'Final Price' ... that were omitted for 26/3/2011" had been resolved by your earlier response supplying those data so that this thread had come to its end, as acknowledged in my response of the 7'th July that the FYI system displays above. My letter of the 14'th August (hand-delivered to preclude "lost in the post" options) concerned something else, unconnected to this sequence, and so your response here could be considered "lost in the internet".

Yours sincerely,

Nicky McLean