SDSS-3 Operations Software Design Log

June 11, 2009

Data Mining

Filed under: Uncategorized — sdssops @ 12:04 am

What can we learn from the data stored in postgres during the past 42 days of archiver running on hub25m?

Actors

test=# select * from actors;
 id__int4 | name__text | major__int4 | minor__int4 |          checksum__text
----------+------------+-------------+-------------+----------------------------------
 0 | hub        |           1 |           2 | 2555484966dff5e737cadd3c14089c22
 1 | mcp        |           1 |           1 | 07e6c56715d3f65f6a47ff4ba641d351
 2 | tcc        |           3 |           2 | 13dcc37fa7c4d0358fa294ff21190cfb
 3 | cmds       |           1 |           0 | 5af3d8b81f593d5f452a26c33e880fb6
 4 | keys       |           0 |           0 |
 5 | perms      |           1 |           1 | 36f6dbcebab2107a5cb2c43ac5f0ec30
(6 rows)

Raw Replies

test=# select * from reply_raw limit 10;
 id__int8 |    tai__flt8     |                                  msg__text
----------+------------------+-----------------------------------------------------------------------------
 0 | 4747665534.86932 | .hub 0 hub i Commanders="client_1","nclient_4"
 1 | 4747665534.87532 | .hub 0 hub i users=
 2 | 4747665567.20724 | .mcp 0 mcp i aliveAt=1240948699
 3 | 4747665581.76263 | .tcc 0 tcc I Modu="exe_BrdTelPos"; Text="2049950 packets sent successfully"
 4 | 4747665629.08049 | .mcp 0 mcp i aliveAt=1240948761
 5 | 4747665631.76203 | .tcc 0 tcc I Modu="exe_BrdTelPos"; Text="2050000 packets sent successfully"
 6 | 4747666020.81695 | .hub 0 hub i Commanders="client_1","nclient_5"
 7 | 4747666020.81952 | .hub 0 hub i users=
 8 |  4747666031.7626 | .tcc 0 tcc I Modu="exe_BrdTelPos"; Text="2050400 packets sent successfully"
 9 | 4747666062.19308 | .mcp 0 mcp i aliveAt=1240949194
(10 rows)

test=# select max(id__int8) from reply_raw;
 max
---------
 8140900
(1 row)

test=# select count(*) from reply_raw;
 count
---------
 8140901
(1 row)

No evidence of missing or duplicate reply IDs, good!

Should make a plot of number of replies binned by hour. Try using int(tai__flt8/3600)…

test=# select round(tai__flt8/3600.)-1318796 as hr,count(*) from reply_raw group by hr;

hourly

This shows the hourly number of raw messages handled over 1000 hours (about 42 days). 36000 corresponds to 10Hz, so the peak rates here are quite high, especially since these are sustained over an hour or more.

  • Why did the peak rate drop by a factor of 2 half way through?
  • The shoulder gradually falls from 200-400 hours. Is this due to aliasing against some sidereal schedule?
  • Why does this query take so long to run? Is better indexing needed or is there competition with the server writing to the tables?

Reply Headers

Count the number of parsed replies per actor:

test=# select name__text,actor_id__int4,count(*) from reply_hdr,actors where reply_hdr.actor_id__int4=actors.id__int4 group by name__text,actor_id__int4;
 name__text | actor_id__int4 |  count
------------+----------------+---------
 tcc        |              2 | 3708666
 mcp        |              1 | 2349048
 keys       |              4 |     174
 perms      |              5 |      18
 cmds       |              3 | 2081844
 hub        |              0 |    1033
(6 rows)

Print hourly traffic by actor:

# select round(tai__flt8/3600.)-1318796 as hr,count(*) from reply_raw,reply_hdr where id__int8=raw_id__int8 and actor_id__int4=2 and tai__flt8/3600.-1318796<100 group by hr;

> plot 'hourlyByActor.dat' using 1:2 with lines title 'TCC','hourlyByActor.dat' using 1:3 with lines title 'MCP','hourlyByActor.dat' using 1:4 with lines title 'CMDS'

hourlyByActor

Print out raw replies that have no corresponding header entry:

test=# select count(*) from reply_raw;
select count(*) from reply_hdr;  count  
---------
 8140901
(1 row)

test=# select count(*) from reply_hdr;
 count  
---------
 8140783
(1 row)

test=# select msg__text from reply_raw left join reply_hdr on id__int8=raw_id__int8 where raw_id__int8 is null limit 200;
 msg__text                                                     
-------------------------------------------------------------------------------------------------------------------
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 unnamed 1 auth : nonce="HFBffTlgP39IdyGGFGG8SXq5NVJwqpmnb2iBL4WE+Ud6v7WtXBN1xDEF87rjoSSWmkyLHeRPrCzsYcAfbFKSMQ=="
 .hub 0 hub i users=
 unnamed 1 auth : loggedIn; cmdrID="APO.Craig"
 .hub 0 hub i users=
 unnamed 1 auth : nonce="ScT54pVAWh+F4E4CrShkEXd4lWn1Eqepo0+DQ1PK/Bpcc9SCAD3C9aaF3mD2jvblnMJW7l0BIPOxb1prGdzfuQ=="
 .hub 0 hub i users=
 unnamed 1 auth : loggedIn; cmdrID="APO.Craig"
 test 1001 keys f keysTxt="local variable 'er' referenced before assignment"
 test 1002 keys f keysTxt="local variable 'er' referenced before assignment"
 test 1003 keys f keysTxt="local variable 'er' referenced before assignment"
 test 1004 keys f keysTxt="local variable 'er' referenced before assignment"
 test 1005 tcc I TAI=4750070587.39
 test 1005 tcc I LST=19.1772; UT1=4750070553.65; UTC_TAI=-34
 test 1005 tcc : Cmd="show time"
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 .hub 0 hub i users=
 None 1001 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1002 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1003 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1004 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1005 tcc I TAI=4750936485.79
 None 1005 tcc I LST=36.9653; UT1=4750936452.04; UTC_TAI=-34
 None 1005 tcc : Cmd="show time"
 .hub 0 hub i users=
 .hub 0 hub i users=
 None 1006 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1007 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1008 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1009 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1010 tcc I TAI=4750942161.01
 None 1010 tcc I LST=60.6768; UT1=4750942127.26; UTC_TAI=-34
 None 1010 tcc : Cmd="show time"
 .hub 0 hub i users=
 .hub 0 hub i users=
 None 1011 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1012 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1013 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1014 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1015 tcc I TAI=4751149403.08
 None 1015 tcc I LST=206.5497; UT1=4751149369.33; UTC_TAI=-34
 None 1015 tcc : Cmd="show time"
 .hub 0 hub i users=
 .hub 0 hub i users=
 None 1001 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1002 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1003 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1004 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1005 tcc I TAI=4751175162.02
 None 1005 tcc I LST=314.1725; UT1=4751175128.27; UTC_TAI=-34
 None 1005 tcc : Cmd="show time"
 .hub 0 hub i users=
 .hub 0 hub i users=
 None 1001 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1002 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1003 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1004 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1005 tcc I TAI=4751178079.67
 None 1005 tcc I LST=326.3626; UT1=4751178045.92; UTC_TAI=-34
 None 1005 tcc : Cmd="show time"
 .hub 0 hub i users=
 .hub 0 hub i users=
 None 1001 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1002 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1003 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1004 keys f keysTxt="local variable 'er' referenced before assignment"
 None 1005 tcc I TAI=4751198864.21
 None 1005 tcc I LST=53.2020; UT1=4751198830.46; UTC_TAI=-34
 None 1005 tcc : Cmd="show time"
 .hub 0 hub i users=
 unnamed 1 auth : nonce="6rIUXXPhIQRset4AQoBLSRDZK+Nf8ZBDHcYS2XgIE2rTA6JSZkwJRTWj2/43i/Rar0f1zO2Fzdhwe45k1il2mg=="
 .hub 0 hub i users=
 unnamed 1 auth : loggedIn; cmdrID="APO.Craig"
 .hub 0 hub i users=
(118 rows)

June 9, 2009

Archiver Postmortem

Filed under: Uncategorized — sdssops @ 10:31 pm

The archiver has been running on hub25m since 28-Apr-09 (41 days). What can we learn from this?

Collect some stats from the info web report:

  • 177 active tables
  • last activity about 16 hours ago
  • 6 active actors:
    • cmds 1.0: 7/8 keywords active, 8.1M updates
    • hub 1.2: 5/5 keywords active, 1.0K updates
    • keys: no dictionary available
    • mcp 1.1: 41/72 keywords active, 2.3M updates
    • perms 1.1: 3/4 keywords active, 8 updates
    • tcc 3.2: 118/171 keywords active, 5.7M updates

Look at the server output files. Since we are running as PID 28070, these should be in /tmp/archiver-28070/:

[dkirkby@hub25m archiver-28070]$ ls -l /tmp/archiver-28070
total 144864
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__cmdactor_15033
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__cmddone_11628
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__cmdr_15266
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__cmdrmid_15150
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__cmdtext_14935
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__cmdtime_15384
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 cmds__newcmd_15509
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 hub__actors_3
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 reply_hdr_119714
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 reply_raw_121066
-rw-rw-r-- 1 dkirkby dkirkby 148186745 Jun  9 18:05 server.log
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__airtemp_1656
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__altdtime_2661
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__altstat_2704
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__axepos_34177
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:33 tcc__axiscmdstate_1087
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 tcc__axiserrcode_1104
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 00:52 tcc__axisinit_638
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 06:24 tcc__axisnoslew_151
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 tcc__axisnotrack_34
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 10:41 tcc__axisstop_334
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 02:40 tcc__azaltdist_229
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__azdtime_2732
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__azstat_18975
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__badaltstatus_1597
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  2 17:01 tcc__badazdtime_1
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__badazstatus_6989
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__badrotstatus_1546
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 11:00 tcc__baduser_61
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__boresight_1959
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__caliboff_1882
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__cmd_5338
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__cmddtime_27739
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__currarcoff_935
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:01 tcc__etext_1068
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 02:20 tcc__expected_59
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:27 tcc__facil_85
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 10:40 tcc__failed_155
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:34 tcc__gcamcmd_1434
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:48 tcc__gcfocus_926
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__gcnfilt_106
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__gcview_107
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__gimctr_84
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__gimlim_84
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__gimscale_84
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__guideoff_1904
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__humidity_1588
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__iimctr_85
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__iimlim_84
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__iimscale_85
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__inst_108
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__instpos_107
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__ipconfig_85
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:01 tcc__lst_6391
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 tcc__modu_66458
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 10:32 tcc__moved_10342
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__moveitems_13692
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:34 tcc__msgslost_324
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objarcoff_2017
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objinstang_1562
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objmag_2580
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objname_2823
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objnetpos_1492
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objoff_2364
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 10:32 tcc__objpm_1789
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objpos_2237
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objsys_2430
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__objzpmpos_2149
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__pressure_1599
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__primactmount_28656
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__primcmdmount_28898
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:05 tcc__primconstrms_29780
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:05 tcc__primdesorient_30058
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__primdesorientage_219
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__primf_bftemp_1629
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__primorient_28353
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__primstatus_218
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 02:40 tcc__ptrefpos_231
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__received_291
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  7 02:22 tcc__rejectedaxiserrcode_102
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__rotdtime_2617
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__rotid_2584
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__rotlim_84
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 10:32 tcc__rotmount_795
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__rotpos_1888
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__rotstat_2635
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:49 tcc__rottype_1926
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 09:57 tcc__rotwrappref_181
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:48 tcc__scalefac_633
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  4 04:25 tcc__schmovetimes_32
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__secactmount_27535
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__seccmdmount_28023
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:05 tcc__secconstrms_29237
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:05 tcc__secdesorient_29504
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__secdesorientage_216
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__secf_bftemp_1612
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:48 tcc__secfocus_971
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__secorient_26804
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__secstatus_216
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__sectrusstemp_1649
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__slewadvtime_1334
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__slewbeg_1251
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:32 tcc__slewduration_1215
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:33 tcc__slewend_957
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  6 08:30 tcc__slewsuperseded_43
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:50 tcc__spiderinstang_2814
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:35 tcc__started_2211
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:01 tcc__tai_8998
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__tccpos_48505
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__tccstatus_59950
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 10:32 tcc__telmount_828
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:12 tcc__text_56561
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__timestamp_1537
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__tlapse_1574
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:06 tcc__trackadvtime_10348
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__useradded_399
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:27 tcc__userdeleted_242
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 11:00 tcc__userinfo_61
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:45 tcc__usernum_959
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:01 tcc__ut1_4254
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 02:01 tcc__utc_tai_3224
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:27 tcc__vmsmsg_84
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 09:52 tcc__warnaltstatus_93
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  8 09:51 tcc__warnazstatus_43
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  1 04:27 tcc__warnrotstatus_42
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__winddir_1549
-rw-rw-r-- 1 dkirkby dkirkby         0 Jun  9 01:52 tcc__windspeed_1557

Table caches are all empty, presumably because they have had plenty of time to be flushed with no recent activity. The log file is about 150Mb in size!

TODO: Be less verbose in logfile

TODO: Implement logfile rotation

There are 127 files here:

  • logfile
  • actors table cache
  • raw table cache
  • reply table cache
  • 123 keyword table caches

How does this reconcile with the 177 active tables? For example, why is there only 1 hub keyword cache file?

Check the server process status:

[dkirkby@hub25m archiver-28070]$ ps -p 28070 -O pmem,pcpu,etime,rss,sz,vsz,wchan
 PID %MEM %CPU     ELAPSED   RSS    SZ    VSZ WCHAN  S TTY          TIME COMMAND
28070  1.1  0.1 41-19:05:59 23096 94259 377036 stext  S ?        01:45:04 python /home/sdss3/products/Linux64/archiver/trunk/bin/archiveServer.p

Explanation of fields:

  • Non-swapped physical memory usage (RSS) is 23Mb, which is 1.1% (%MEM) of the host’s available physical memory
  • Size in physical pages of the core image of the process (SZ) is 94,259 (how big is a physical page??)
  • Virtual memory size of the process (VSZ) is 377Mb
  • Used an average of 0.1% of the host CPU since it started, 41 days 19 hours ago (ELAPSED)
  • Total CPU usage (system+user) is 1 hour 45 mins
  • Process is sleeping on the “stext” kernel function (WCHAN)

What about the postgres server processes? Follow these instructions:

[dkirkby@hub25m archiver-28070]$ ps -u postgres -O pmem,pcpu,etime,rss,sz,vsz,wchan
 PID %MEM %CPU     ELAPSED   RSS    SZ    VSZ WCHAN  S TTY          TIME COMMAND
 2800  0.1  0.0 45-19:15:26  4024 27575 110300 -      S ?        00:00:09 /usr/local/pgsql/bin/postmaster -D /usr/local/pgsql/data
 2878  1.6  0.0 45-19:15:25 34340 27608 110432 -      S ?        00:00:14 postgres: writer process
 2879  0.0  0.0 45-19:15:25   968 27575 110300 -      S ?        00:00:00 postgres: wal writer process
 2880  0.0  0.0 45-19:15:25  1232 27666 110664 -      S ?        00:00:04 postgres: autovacuum launcher process
 2881  0.0  0.0 45-19:15:25   980 18052  72208 -      S ?        00:00:38 postgres: stats collector process
28078  1.8  0.0 41-19:44:57 38012 28163 112652 -      S ?        00:02:42 postgres: testing test 127.0.0.1(57193) idle
28079  1.8  0.0 41-19:44:56 38008 28146 112584 -      S ?        00:02:41 postgres: testing test 127.0.0.1(57194) idle
28080  1.8  0.0 41-19:44:55 37976 28160 112640 -      S ?        00:02:42 postgres: testing test 127.0.0.1(57195) idle
28662  1.8  0.0 41-18:24:51 37940 28134 112536 -      S ?        00:02:40 postgres: testing test 127.0.0.1(43754) idle
28664  1.8  0.0 41-18:24:51 37956 28141 112564 -      S ?        00:02:42 postgres: testing test 127.0.0.1(43755) idle

The last 5 processes are per client connection. The twisted ADBAPI ConnectionPool maintains 3-5 (max) connections, so this makes sense. Two of these processes started about 80 mins later than the first three. Each connection’s handler has a relatively large memory footprint (38Mb RSS) but has used very little CPU (<3 mins over 42 days).

The postgres stats collector process is running, so try to talk to that… Stats are recorded in special database tables (can I read these?) with predefined “views” (?)

Postgres “views” are described here. Try the following (why doesn’t psql ask for a password??):

[dkirkby@hub25m archiver-28070]$ /usr/local/pgsql/bin/psql -d test -U testing
Welcome to psql 8.3.7, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
 \h for help with SQL commands
 \? for help with psql commands
 \g or terminate with semicolon to execute query
 \q to quit

test=# select * from pg_stat_activity;
 datid | datname | procpid | usesysid | usename |         current_query          | waiting |          xact_start           |          query_start          |         backend_start         | client_addr | client_port
-------+---------+---------+----------+---------+--------------------------------+---------+-------------------------------+-------------------------------+-------------------------------+-------------+-------------
 16384 | test    |   28078 |    16385 | testing | <IDLE>                         | f       |                               | 2009-06-08 20:12:18.944562-06 | 2009-04-28 18:59:48.884767-06 | 127.0.0.1   |       57193
 16384 | test    |   28079 |    16385 | testing | <IDLE>                         | f       |                               | 2009-06-08 20:12:17.884422-06 | 2009-04-28 18:59:49.884668-06 | 127.0.0.1   |       57194
 16384 | test    |   28080 |    16385 | testing | <IDLE>                         | f       |                               | 2009-06-08 20:12:16.884154-06 | 2009-04-28 18:59:50.884581-06 | 127.0.0.1   |       57195
 16384 | test    |   28662 |    16385 | testing | <IDLE>                         | f       |                               | 2009-06-08 20:12:15.884124-06 | 2009-04-28 20:19:54.778255-06 | 127.0.0.1   |       43754
 16384 | test    |   28664 |    16385 | testing | <IDLE>                         | f       |                               | 2009-06-08 20:12:19.885119-06 | 2009-04-28 20:19:54.781082-06 | 127.0.0.1   |       43755
 16384 | test    |   32060 |    16385 | testing | select * from pg_stat_activity | f       | 2009-06-09 15:09:10.366786-06 | 2009-06-09 15:09:10.366786-06 | 2009-06-09 15:08:15.694922-06 |             |          -1
 : ;
(6 rows)

This agrees with the process info above (the extra connection is for my psql session). Try some other views:

test=# select * from pg_stat_database;
 datid |   datname   | numbackends | xact_commit | xact_rollback | blks_read | blks_hit  | tup_returned | tup_fetched | tup_inserted | tup_updated | tup_deleted
-------+-------------+-------------+-------------+---------------+-----------+-----------+--------------+-------------+--------------+-------------+-------------
     1 | template1   |           0 |           0 |             0 |         0 |         0 |            0 |           0 |            0 |           0 |           0
 11510 | template0   |           0 |           0 |             0 |         0 |         0 |            0 |           0 |            0 |           0 |           0
 11511 | postgres    |           0 |      192965 |             3 |       576 |  11769109 |     68086973 |     2606494 |            0 |          12 |           0
 16384 | test        |           6 |     1337094 |            16 |   1535688 | 203852699 |    145118156 |     3275594 |     32397971 |       13438 |           0
 17374 | sdss3       |           0 |       96985 |             0 |       365 |   5917192 |     34231867 |     1310859 |            0 |           0 |           0
 17375 | apo_platedb |           0 |       99245 |           552 |      4143 |  10081099 |     68146278 |     1622814 |        71814 |       49545 |        1990
(6 rows)

The tup_inserted column reports the total number of rows inserted (including rows inserted before this server process started). For the archiver, it is at 32,397,971. What is tup_returned and why is it so high? Who is responsible for the 3.3M fetches and 13K updates?

test=# select * from pg_stat_user_tables;
 relid | schemaname |           relname            | seq_scan | seq_tup_read | idx_scan | idx_tup_fetch | n_tup_ins | n_tup_upd | n_tup_del | n_tup_hot_upd | n_live_tup | n_dead_tup | last_vacuum | last_autovacuum | last_analyze | last_autoanalyze
-------+------------+------------------------------+----------+--------------+----------+---------------+-----------+-----------+-----------+---------------+------------+------------+-------------+-----------------+--------------+-------------------------------
 16983 | public     | tcc__winddir                 |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-06 22:56:41.928523-06
 16741 | public     | tcc__primconstrms            |        7 |           79 |        0 |             0 |     33941 |         0 |         0 |             0 |      33941 |          0 |             |                 |              | 2009-06-07 20:23:05.824867-06
 16620 | public     | tcc__objsys                  |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-08 03:25:46.350949-06
 16552 | public     | tcc__gcfocus                 |        7 |            7 |        0 |             0 |     32909 |         0 |         0 |             0 |      32909 |          0 |             |                 |              | 2009-06-04 22:38:58.044076-06
 17077 | public     | mcp__needsemaphore           |        1 |            0 |        0 |             0 |       160 |         0 |         0 |             0 |        160 |          0 |             |                 |              | 2009-05-13 20:27:30.360038-06
 16475 | public     | tcc__etext                   |        7 |           19 |        0 |             0 |      1342 |         0 |         0 |             0 |       1342 |          0 |             |                 |              | 2009-06-08 20:02:34.738122-06
 17108 | public     | tcc__rotwrappref             |        1 |            0 |        0 |             0 |       239 |         0 |         0 |             0 |        239 |          0 |             |                 |              | 2009-06-08 02:51:37.496345-06
 17171 | public     | tcc__slewsuperseded          |        1 |            0 |        0 |             0 |        51 |         0 |         0 |             0 |         51 |          0 |             |                 |              | 2009-06-06 02:30:40.822591-06
 16668 | public     | tcc__objnetpos               |        7 |           15 |        0 |             0 |     70442 |         0 |         0 |             0 |      70442 |          0 |             |                 |              | 2009-06-06 21:08:31.927628-06
 17313 | public     | mcp__plcversion              |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16935 | public     | mcp__nelamp                  |        3 |            4 |        0 |             0 |       197 |         0 |         0 |             0 |        197 |          0 |             |                 |              | 2009-05-13 04:48:58.665521-06
 17059 | public     | tcc__axisstop                |        3 |            1 |        0 |             0 |       366 |         0 |         0 |             0 |        366 |          0 |             |                 |              | 2009-06-04 03:49:07.545043-06
 16938 | public     | mcp__nelampcommandedon       |        3 |            4 |        0 |             0 |       193 |         0 |         0 |             0 |        193 |          0 |             |                 |              | 2009-05-13 18:53:21.152422-06
 16875 | public     | tcc__iimscale                |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.316256-06
 16986 | public     | tcc__tlapse                  |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-06 22:56:43.973815-06
 17176 | public     | tcc__rejectedaxiserrcode     |        1 |            0 |        0 |             0 |       164 |         0 |         0 |             0 |        164 |          0 |             |                 |              | 2009-05-13 05:05:00.233784-06
 16880 | public     | tcc__iimctr                  |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.32073-06
 16480 | public     | tcc__tai                     |        7 |           10 |        0 |             0 |     35443 |         0 |         0 |             0 |      35443 |          0 |             |                 |              | 2009-06-04 03:41:07.119444-06
 16955 | public     | tcc__airtemp                 |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-06 01:56:38.36679-06
 16436 | public     | cmds__cmddone                |        7 |         2809 |        0 |             0 |    876636 |         0 |         0 |             0 |     876242 |          0 |             |                 |              | 2009-06-01 02:51:05.716353-06
 17356 | public     | mcp__ffscommanded            |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16573 | public     | tcc__gcview                  |        7 |            8 |        0 |             0 |       117 |         0 |         0 |             0 |        117 |          0 |             |                 |              | 2009-06-03 17:57:09.854948-06
 17128 | public     | tcc__telmount                |        1 |            0 |        0 |             0 |     32544 |         0 |         0 |             0 |      32544 |          0 |             |                 |              | 2009-06-04 23:51:05.318318-06
 17244 | public     | mcp__ffsselected             |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17366 | public     | mcp__text                    |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16386 | public     | reply_raw                    |       12 |        24209 |        0 |             0 |   8140779 |         0 |         0 |             0 |    8147320 |          0 |             |                 |              | 2009-06-02 21:13:35.418578-06
 17308 | public     | mcp__minencodermismatch      |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17158 | public     | mcp__ffsopenfailed           |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16640 | public     | tcc__boresight               |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-06 00:35:40.827762-06
 16734 | public     | tcc__slewbeg                 |        7 |            6 |        0 |             0 |      1481 |         0 |         0 |             0 |       1481 |          0 |             |                 |              | 2009-06-06 22:07:35.743777-06
 16583 | public     | tcc__gcnfilt                 |        7 |            8 |        0 |             0 |       117 |         0 |         0 |             0 |        117 |          0 |             |                 |              | 2009-06-03 17:57:09.793692-06
 16897 | public     | tcc__gimctr                  |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.339633-06
 16792 | public     | tcc__altstat                 |        3 |          437 |        0 |             0 |    127561 |         0 |         0 |             0 |     127561 |          0 |             |                 |              | 2009-06-03 03:56:49.084789-06
 16426 | public     | tcc__modu                    |       11 |          866 |        0 |             0 |     86339 |         0 |         0 |             0 |      86339 |          0 |             |                 |              | 2009-06-07 08:36:52.180351-06
 16807 | public     | tcc__rotstat                 |        3 |          423 |        0 |             0 |    127019 |         0 |         0 |             0 |     127019 |          0 |             |                 |              | 2009-06-03 04:17:51.084144-06
 16854 | public     | tcc__seccmdmount             |        3 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-03 03:17:44.301925-06
 17090 | public     | tcc__axisnoslew              |        1 |            0 |        0 |             0 |       226 |         0 |         0 |             0 |        226 |          0 |             |                 |              | 2009-05-15 02:29:40.207621-06
 16777 | public     | tcc__badazstatus             |        3 |          407 |        0 |             0 |    124659 |         0 |         0 |             0 |     124659 |          0 |             |                 |              | 2009-06-03 18:59:16.680246-06
 16844 | public     | tcc__primactmount            |        3 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-06 21:52:42.050244-06
 17184 | public     | tcc__warnaltstatus           |        1 |            0 |        0 |             0 |       214 |         0 |         0 |             0 |        214 |          0 |             |                 |              | 2009-06-01 20:22:43.31796-06
 16660 | public     | tcc__rottype                 |        7 |            5 |        0 |             0 |    100266 |         0 |         0 |             0 |     100266 |          0 |             |                 |              | 2009-06-06 03:33:48.400504-06
 16925 | public     | mcp__fflamp                  |        3 |            4 |        0 |             0 |       551 |         0 |         0 |             0 |        551 |          0 |             |                 |              | 2009-05-13 01:05:37.033948-06
 17000 | public     | tcc__timestamp               |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-06 22:56:45.944251-06
 16907 | public     | tcc__rotlim                  |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.35651-06
 17298 | public     | mcp__goodfiducialversions    |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17318 | public     | mcp__mcpversion              |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16516 | public     | mcp__command                 |        7 |         2685 |        0 |             0 |    553750 |         0 |         0 |             0 |     553842 |          0 |             |                 |              | 2009-05-14 14:41:35.281016-06
 16723 | public     | tcc__moveitems               |        7 |            6 |        0 |             0 |     68262 |         0 |         0 |             0 |      68262 |          0 |             |                 |              | 2009-06-08 02:20:36.670382-06
 16980 | public     | tcc__humidity                |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-04 03:25:06.327849-06
 17153 | public     | mcp__ffsmoved                |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16557 | public     | tcc__started                 |        7 |           15 |        0 |             0 |    106274 |         0 |         0 |             0 |     106274 |          0 |             |                 |              | 2009-06-06 02:10:40.677567-06
 16814 | public     | tcc__badrotstatus            |        3 |          407 |        0 |             0 |    121452 |         0 |         0 |             0 |     121452 |          0 |             |                 |              | 2009-06-03 20:50:27.511216-06
 16562 | public     | tcc__inst                    |        7 |            8 |        0 |             0 |       117 |         0 |         0 |             0 |        117 |          0 |             |                 |              | 2009-06-03 17:56:09.772041-06
 16657 | public     | tcc__caliboff                |        7 |            7 |        0 |             0 |     68536 |         0 |         0 |             0 |      68536 |          0 |             |                 |              | 2009-06-06 01:42:41.066993-06
 16885 | public     | tcc__iimlim                  |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.3324-06
 17199 | public     | tcc__warnazstatus            |        1 |            0 |        0 |             0 |       110 |         0 |         0 |             0 |        110 |          0 |             |                 |              | 2009-06-07 22:04:09.651133-06
 17098 | public     | tcc__schmovetimes            |        1 |            0 |        0 |             0 |        34 |         0 |         0 |             0 |         34 |          0 |             |                 |              |
 16663 | public     | tcc__currarcoff              |        7 |           15 |        0 |             0 |      3047 |         0 |         0 |             0 |       3047 |          0 |             |                 |              | 2009-06-05 00:00:05.881382-06
 16683 | public     | tcc__spiderinstang           |        7 |           15 |        0 |             0 |    197729 |         0 |         0 |             0 |     197729 |          0 |             |                 |              | 2009-06-06 01:13:37.054459-06
 17336 | public     | mcp__lampscommanded          |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16780 | public     | tcc__azstat                  |        3 |          423 |        0 |             0 |    129470 |         0 |         0 |             0 |     129470 |          0 |             |                 |              | 2009-06-03 03:08:44.352749-06
 16767 | public     | tcc__tccpos                  |        3 |          527 |        0 |             0 |    270968 |         0 |         0 |             0 |     270968 |          0 |             |                 |              | 2009-06-04 21:17:52.402369-06
 16789 | public     | tcc__azdtime                 |        3 |          423 |        0 |             0 |    129470 |         0 |         0 |             0 |     129470 |          0 |             |                 |              | 2009-06-03 21:23:31.21781-06
 17796 | public     | tcc__superseded              |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17809 | public     | hub__actors                  |        1 |            0 |        0 |             0 |         2 |         0 |         0 |             0 |          2 |          0 |             |                 |              |
 16761 | public     | tcc__tccstatus               |        3 |          527 |        0 |             0 |    270968 |         0 |         0 |             0 |     270968 |          0 |             |                 |              | 2009-06-04 03:05:05.271948-06
 17223 | public     | perms__authlist              |        1 |            0 |        0 |             0 |         2 |         0 |         0 |             0 |          2 |          0 |             |                 |              |
 16444 | public     | cmds__cmdtime                |        7 |         4457 |        0 |             0 |   1205099 |         0 |         0 |             0 |    1207655 |          0 |             |                 |              | 2009-06-01 22:32:09.617067-06
 16501 | public     | tcc__utc_tai                 |        7 |           10 |        0 |             0 |     35443 |         0 |         0 |             0 |      35443 |          0 |             |                 |              | 2009-06-07 23:36:18.926059-06
 16508 | public     | mcp__userid                  |        7 |           78 |        0 |             0 |       878 |         0 |         0 |             0 |        878 |          0 |             |                 |              | 2009-05-14 10:49:54.329358-06
 16729 | public     | tcc__slewduration            |        7 |            6 |        0 |             0 |      1481 |         0 |         0 |             0 |       1481 |          0 |             |                 |              | 2009-06-06 22:07:36.415961-06
 16772 | public     | tcc__axepos                  |        3 |          527 |        0 |             0 |    270968 |         0 |         0 |             0 |     270968 |          0 |             |                 |              | 2009-06-05 04:31:34.645188-06
 16751 | public     | tcc__secdesorient            |        7 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-06 21:49:38.412546-06
 18299 | public     | tcc__badazdtime              |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16834 | public     | tcc__cmddtime                |        3 |          158 |        0 |             0 |     67882 |         0 |         0 |             0 |      67882 |          0 |             |                 |              | 2009-06-03 05:11:55.795873-06
 17194 | public     | mcp__rotmaxaccrequested      |        1 |            0 |        0 |             0 |        38 |         0 |         0 |             0 |         38 |          0 |             |                 |              |
 16637 | public     | tcc__objoff                  |        7 |            7 |        0 |             0 |    101506 |         0 |         0 |             0 |     101506 |          0 |             |                 |              | 2009-06-06 04:08:03.71199-06
 17148 | public     | tcc__userinfo                |        1 |            0 |        0 |             0 |        94 |         0 |         0 |             0 |         94 |          0 |             |                 |              | 2009-05-18 06:01:17.789782-06
 17280 | public     | mcp__azfiducialversion       |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17204 | public     | hub__users                   |        1 |            0 |        0 |             0 |         2 |         0 |         0 |             0 |          2 |          0 |             |                 |              |
 17327 | public     | mcp__cwstatus                |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17212 | public     | hub__user                    |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17341 | public     | mcp__azmson                  |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16652 | public     | tcc__guideoff                |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-06 00:35:48.135865-06
 16886 | public     | tcc__gimscale                |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.336272-06
 16646 | public     | tcc__objarcoff               |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-06 00:35:43.461865-06
 16697 | public     | mcp__commandin               |        7 |         8478 |        0 |             0 |   1755760 |         0 |         0 |             0 |    1757290 |          0 |             |                 |              | 2009-05-14 03:41:20.939866-06
 16713 | public     | tcc__axiscmdstate            |        7 |           21 |        0 |             0 |      5429 |         0 |         0 |             0 |       5429 |          0 |             |                 |              | 2009-06-08 02:58:39.019249-06
 16453 | public     | cmds__cmdr                   |        7 |         4457 |        0 |             0 |   1205099 |         0 |         0 |             0 |    1204508 |          0 |             |                 |              | 2009-06-01 22:20:03.040484-06
 16493 | public     | tcc__lst                     |        7 |           10 |        0 |             0 |     35443 |         0 |         0 |             0 |      35443 |          0 |             |                 |              | 2009-06-06 01:03:40.622075-06
 17028 | public     | tcc__secstatus               |        3 |            2 |        0 |             0 |       249 |         0 |         0 |             0 |        249 |          0 |             |                 |              | 2009-06-06 18:52:17.767691-06
 16917 | public     | tcc__gcamcmd                 |        3 |            2 |        0 |             0 |     36025 |         0 |         0 |             0 |      36025 |          0 |             |                 |              | 2009-06-07 22:32:38.494962-06
 16682 | public     | tcc__objinstang              |        7 |           15 |        0 |             0 |     68488 |         0 |         0 |             0 |      68488 |          0 |             |                 |              | 2009-06-07 04:22:27.020162-06
 16795 | public     | tcc__badaltstatus            |        3 |          424 |        0 |             0 |    122404 |         0 |         0 |             0 |     122404 |          0 |             |                 |              | 2009-06-03 09:18:20.084785-06
 16798 | public     | tcc__altdtime                |        3 |          437 |        0 |             0 |    127561 |         0 |         0 |             0 |     127561 |          0 |             |                 |              | 2009-06-03 03:56:50.115251-06
 16441 | public     | cmds__newcmd                 |        7 |         4457 |        0 |             0 |   1205099 |         0 |         0 |             0 |    1205724 |          0 |             |                 |              | 2009-06-01 22:32:02.252852-06
 16472 | public     | cmds__cmdtext                |        7 |         4457 |        0 |             0 |   1205099 |         0 |         0 |             0 |    1206345 |          0 |             |                 |              | 2009-06-04 01:04:58.542682-06
 17049 | public     | tcc__userdeleted             |        3 |            1 |        0 |             0 |       294 |         0 |         0 |             0 |        294 |          0 |             |                 |              | 2009-06-01 04:48:11.951696-06
 17249 | public     | mcp__uvlampcommandedon       |        1 |            0 |        0 |             0 |         3 |         0 |         0 |             0 |          3 |          0 |             |                 |              |
 16524 | public     | tcc__usernum                 |        7 |           32 |        0 |             0 |      2321 |         0 |         0 |             0 |       2321 |          0 |             |                 |              | 2009-06-04 18:41:34.732719-06
 17361 | public     | mcp__azmaxaccrequested       |        1 |            0 |        0 |             0 |         2 |         0 |         0 |             0 |          2 |          0 |             |                 |              |
 16588 | public     | tcc__vmsmsg                  |        7 |            9 |        0 |             0 |       351 |         0 |         0 |             0 |        351 |          0 |             |                 |              | 2009-06-02 05:03:33.926494-06
 16625 | public     | tcc__objpos                  |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-08 03:26:02.286923-06
 17221 | public     | perms__actors                |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16835 | public     | tcc__primcmdmount            |        3 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-06 21:52:36.213869-06
 16970 | public     | tcc__secf_bftemp             |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-04 03:25:05.508712-06
 17189 | public     | tcc__warnrotstatus           |        1 |            0 |        0 |             0 |        92 |         0 |         0 |             0 |         92 |          0 |             |                 |              | 2009-05-05 03:27:29.949377-06
 16945 | public     | mcp__hgcdlamp                |        3 |            4 |        0 |             0 |       191 |         0 |         0 |             0 |        191 |          0 |             |                 |              | 2009-05-13 18:57:21.58376-06
 16534 | public     | tcc__cmd                     |        7 |           65 |        0 |             0 |    310224 |         0 |         0 |             0 |     310224 |          0 |             |                 |              | 2009-06-07 21:13:25.450552-06
 16529 | public     | tcc__useradded               |        7 |           15 |        0 |             0 |       451 |         0 |         0 |             0 |        451 |          0 |             |                 |              | 2009-06-03 17:49:09.023096-06
 16869 | public     | tcc__ipconfig                |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.35999-06
 17041 | public     | tcc__axisinit                |        3 |            2 |        0 |             0 |       778 |         0 |         0 |             0 |        778 |          0 |             |                 |              | 2009-06-04 23:14:01.256108-06
 17326 | public     | mcp__cwpositions             |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16718 | public     | tcc__slewadvtime             |        7 |            6 |        0 |             0 |      1481 |         0 |         0 |             0 |       1481 |          0 |             |                 |              | 2009-06-06 22:07:36.337009-06
 17143 | public     | tcc__baduser                 |        1 |            0 |        0 |             0 |        94 |         0 |         0 |             0 |         94 |          0 |             |                 |              | 2009-05-18 06:01:17.756818-06
 16410 | public     | actors                       |       16 |           36 |        0 |             0 |         6 |         0 |         0 |             0 |          6 |          0 |             |                 |              |
 16671 | public     | tcc__rotpos                  |        7 |            5 |        0 |             0 |    100266 |         0 |         0 |             0 |     100266 |          0 |             |                 |              | 2009-06-07 21:42:11.446509-06
 17033 | public     | tcc__axiserrcode             |        3 |            7 |        0 |             0 |      2642 |         0 |         0 |             0 |       2642 |          0 |             |                 |              | 2009-06-05 03:40:27.362753-06
 16817 | public     | tcc__rotid                   |        3 |          423 |        0 |             0 |    127019 |         0 |         0 |             0 |     127019 |          0 |             |                 |              | 2009-06-03 04:17:52.900382-06
 17021 | public     | tcc__secdesorientage         |        3 |            2 |        0 |             0 |       249 |         0 |         0 |             0 |        249 |          0 |             |                 |              | 2009-06-06 18:52:17.607007-06
 17346 | public     | mcp__altmson                 |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17252 | public     | mcp__whtlampcommandedon      |        1 |            0 |        0 |             0 |         3 |         0 |         0 |             0 |          3 |          0 |             |                 |              |
 17299 | public     | mcp__msonmaxcorrection       |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16591 | public     | tcc__facil                   |        7 |            9 |        0 |             0 |       351 |         0 |         0 |             0 |        351 |          0 |             |                 |              | 2009-06-02 05:03:33.842386-06
 16900 | public     | tcc__gimlim                  |        3 |            2 |        0 |             0 |        90 |         0 |         0 |             0 |         90 |          0 |             |                 |              | 2009-05-18 19:34:37.345626-06
 16930 | public     | mcp__fflampcommandedon       |        3 |            4 |        0 |             0 |       551 |         0 |         0 |             0 |        551 |          0 |             |                 |              | 2009-05-13 01:05:37.075223-06
 16754 | public     | tcc__secconstrms             |        7 |           79 |        0 |             0 |     33941 |         0 |         0 |             0 |      33941 |          0 |             |                 |              | 2009-06-07 20:24:02.730173-06
 16958 | public     | tcc__primf_bftemp            |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-04 03:25:06.697511-06
 16859 | public     | tcc__secactmount             |        3 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-07 23:40:25.719849-06
 16864 | public     | tcc__secorient               |        3 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-08 00:26:26.756592-06
 16420 | public     | tcc__text                    |       11 |          901 |        0 |             0 |    170478 |         0 |         0 |             0 |     170478 |          0 |             |                 |              | 2009-06-07 20:53:11.261788-06
 17054 | public     | mcp__rotbadfiducialdelta     |        3 |            1 |        0 |             0 |       522 |         0 |         0 |             0 |        522 |          0 |             |                 |              | 2009-05-13 20:23:30.001655-06
 17283 | public     | mcp__rotfiducialversion      |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16829 | public     | tcc__trackadvtime            |        3 |          827 |        0 |             0 |    394638 |         0 |         0 |             0 |     394638 |          0 |             |                 |              | 2009-06-04 03:21:07.57704-06
 17082 | public     | tcc__axisnotrack             |        1 |            0 |        0 |             0 |        36 |         0 |         0 |             0 |         36 |          0 |             |                 |              |
 16975 | public     | tcc__pressure                |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-04 03:25:05.910841-06
 17801 | public     | hub__text                    |        1 |            0 |        0 |             0 |         2 |         0 |         0 |             0 |          2 |          0 |             |                 |              |
 17138 | public     | tcc__msgslost                |        1 |            0 |        0 |             0 |       681 |         0 |         0 |             0 |        681 |          0 |             |                 |              | 2009-06-02 19:31:58.543574-06
 16995 | public     | tcc__windspeed               |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-06 22:56:47.773618-06
 17269 | public     | mcp__instrumentnum           |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16912 | public     | tcc__slewend                 |        3 |            6 |        0 |             0 |      1426 |         0 |         0 |             0 |       1426 |          0 |             |                 |              | 2009-06-04 04:49:13.326452-06
 16950 | public     | mcp__hgcdlampcommandedon     |        3 |            4 |        0 |             0 |       191 |         0 |         0 |             0 |        191 |          0 |             |                 |              | 2009-05-13 18:57:21.557698-06
 16568 | public     | tcc__instpos                 |        7 |            8 |        0 |             0 |       117 |         0 |         0 |             0 |        117 |          0 |             |                 |              | 2009-06-03 17:57:09.853515-06
 17791 | public     | tcc__exptime                 |        1 |            0 |        0 |             0 |         8 |         0 |         0 |             0 |          8 |          0 |             |                 |              |
 16961 | public     | tcc__sectrusstemp            |        3 |            4 |        0 |             0 |     34133 |         0 |         0 |             0 |      34133 |          0 |             |                 |              | 2009-06-06 01:56:37.850268-06
 17118 | public     | tcc__azaltdist               |        1 |            0 |        0 |             0 |       237 |         0 |         0 |             0 |        237 |          0 |             |                 |              | 2009-06-07 20:39:06.317946-06
 16404 | public     | hub__commanders              |       11 |           13 |        0 |             0 |         9 |         0 |         0 |             0 |          9 |          0 |             |                 |              |
 17072 | public     | mcp__ffscommandedon          |        1 |            0 |        0 |             0 |       502 |         0 |         0 |             0 |        502 |          0 |             |                 |              | 2009-05-14 02:20:04.725829-06
 16449 | public     | cmds__cmdactor               |        7 |         4457 |        0 |             0 |   1205099 |         0 |         0 |             0 |    1202938 |          0 |             |                 |              | 2009-06-01 22:29:01.804393-06
 17067 | public     | mcp__ffsstatus               |        1 |            0 |        0 |             0 |       502 |         0 |         0 |             0 |        502 |          0 |             |                 |              | 2009-05-14 02:20:04.717098-06
 16630 | public     | tcc__objzpmpos               |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-08 01:13:29.394381-06
 16820 | public     | tcc__rotdtime                |        3 |          423 |        0 |             0 |    127019 |         0 |         0 |             0 |     127019 |          0 |             |                 |              | 2009-06-03 04:17:51.996615-06
 16746 | public     | tcc__primdesorient           |        7 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-06 01:38:36.206038-06
 17133 | public     | tcc__rotmount                |        1 |            0 |        0 |             0 |     32544 |         0 |         0 |             0 |      32544 |          0 |             |                 |              | 2009-06-04 23:51:05.72085-06
 17123 | public     | tcc__moved                   |        1 |            0 |        0 |             0 |     66781 |         0 |         0 |             0 |      66781 |          0 |             |                 |              | 2009-06-04 03:37:06.855346-06
 16615 | public     | tcc__objmag                  |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-08 03:11:43.193813-06
 17113 | public     | tcc__ptrefpos                |        1 |            0 |        0 |             0 |       237 |         0 |         0 |             0 |        237 |          0 |             |                 |              | 2009-06-07 20:39:06.288525-06
 16542 | public     | tcc__scalefac                |        7 |           11 |        0 |             0 |      1349 |         0 |         0 |             0 |       1349 |          0 |             |                 |              | 2009-06-06 00:30:31.531405-06
 17351 | public     | mcp__rotmson                 |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17010 | public     | tcc__primstatus              |        3 |            2 |        0 |             0 |       249 |         0 |         0 |             0 |        249 |          0 |             |                 |              | 2009-06-06 18:52:17.579657-06
 17103 | public     | tcc__objpm                   |        1 |            0 |        0 |             0 |     66994 |         0 |         0 |             0 |      66994 |          0 |             |                 |              | 2009-06-04 03:25:07.983117-06
 16547 | public     | tcc__secfocus                |        7 |           10 |        0 |             0 |     33977 |         0 |         0 |             0 |      33977 |          0 |             |                 |              | 2009-06-07 21:11:08.819183-06
 16463 | public     | cmds__cmdrmid                |        7 |         4457 |        0 |             0 |   1205099 |         0 |         0 |             0 |    1205297 |          0 |             |                 |              | 2009-06-01 22:23:01.895879-06
 17015 | public     | tcc__received                |        3 |            6 |        0 |             0 |       875 |         0 |         0 |             0 |        875 |          0 |             |                 |              | 2009-06-06 18:52:17.755905-06
 16609 | public     | tcc__objname                 |        7 |            7 |        0 |             0 |     67960 |         0 |         0 |             0 |      67960 |          0 |             |                 |              | 2009-06-08 04:20:46.428065-06
 17259 | public     | mcp__saddleismounted         |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16600 | public     | tcc__failed                  |        7 |            3 |        0 |             0 |       229 |         0 |         0 |             0 |        229 |          0 |             |                 |              | 2009-06-01 18:59:35.261864-06
 16849 | public     | tcc__primorient              |        3 |           81 |        0 |             0 |     34190 |         0 |         0 |             0 |      34190 |          0 |             |                 |              | 2009-06-06 21:53:45.542542-06
 17274 | public     | mcp__altfiducialversion      |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16692 | public     | mcp__controllamps            |        7 |           25 |        0 |             0 |      1314 |         0 |         0 |             0 |       1314 |          0 |             |                 |              | 2009-05-14 18:35:54.271096-06
 17220 | public     | perms__programs              |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 17163 | public     | tcc__expected                |        1 |            0 |        0 |             0 |        72 |         0 |         0 |             0 |         72 |          0 |             |                 |              | 2009-05-26 18:36:14.994264-06
 17264 | public     | mcp__instrumentnumconsistent |        1 |            0 |        0 |             0 |         1 |         0 |         0 |             0 |          1 |          0 |             |                 |              |
 16700 | public     | mcp__semaphoreowner          |        7 |            1 |        0 |             0 |       669 |         0 |         0 |             0 |        669 |          0 |             |                 |              | 2009-05-13 02:09:43.220862-06
 16389 | public     | reply_hdr                    |       12 |        24196 |        0 |             0 |   8140665 |         0 |         0 |             0 |    8149768 |          0 |             |                 |              | 2009-06-02 21:22:30.833751-06
 16494 | public     | tcc__ut1                     |        7 |           10 |        0 |             0 |     35443 |         0 |         0 |             0 |      35443 |          0 |             |                 |              | 2009-06-08 01:05:34.578143-06
 17005 | public     | tcc__primdesorientage        |        3 |            2 |        0 |             0 |       249 |         0 |         0 |             0 |        249 |          0 |             |                 |              | 2009-06-06 18:52:16.773582-06
(177 rows)

test=# select * from pg_stat_user_indexes;
 relid | indexrelid | schemaname |           relname            |           indexrelname            | idx_scan | idx_tup_read | idx_tup_fetch
-------+------------+------------+------------------------------+-----------------------------------+----------+--------------+---------------
 16386 |      16398 | public     | reply_raw                    | reply_raw_pkey                    |        0 |            0 |             0
 16389 |      16400 | public     | reply_hdr                    | reply_hdr_pkey                    |        0 |            0 |             0
 16389 |      16402 | public     | reply_hdr                    | reply_hdr_actor_id__int4          |        0 |            0 |             0
 16386 |      16403 | public     | reply_raw                    | reply_raw_tai__flt8               |        0 |            0 |             0
 16404 |      16416 | public     | hub__commanders              | hub__commanders_pkey              |        0 |            0 |             0
 16410 |      16418 | public     | actors                       | actors_pkey                       |        0 |            0 |             0
 16420 |      16432 | public     | tcc__text                    | tcc__text_pkey                    |        0 |            0 |             0
 16426 |      16434 | public     | tcc__modu                    | tcc__modu_pkey                    |        0 |            0 |             0
 16436 |      16439 | public     | cmds__cmddone                | cmds__cmddone_pkey                |        0 |            0 |             0
 16441 |      16447 | public     | cmds__newcmd                 | cmds__newcmd_pkey                 |        0 |            0 |             0
 16444 |      16452 | public     | cmds__cmdtime                | cmds__cmdtime_pkey                |        0 |            0 |             0
 16453 |      16468 | public     | cmds__cmdr                   | cmds__cmdr_pkey                   |        0 |            0 |             0
 16501 |      16504 | public     | tcc__utc_tai                 | tcc__utc_tai_pkey                 |        0 |            0 |             0
 16516 |      16522 | public     | mcp__command                 | mcp__command_pkey                 |        0 |            0 |             0
 16534 |      16540 | public     | tcc__cmd                     | tcc__cmd_pkey                     |        0 |            0 |             0
 16552 |      16555 | public     | tcc__gcfocus                 | tcc__gcfocus_pkey                 |        0 |            0 |             0
 16583 |      16586 | public     | tcc__gcnfilt                 | tcc__gcnfilt_pkey                 |        0 |            0 |             0
 16600 |      16603 | public     | tcc__failed                  | tcc__failed_pkey                  |        0 |            0 |             0
 16463 |      16466 | public     | cmds__cmdrmid                | cmds__cmdrmid_pkey                |        0 |            0 |             0
 16449 |      16470 | public     | cmds__cmdactor               | cmds__cmdactor_pkey               |        0 |            0 |             0
 16472 |      16489 | public     | cmds__cmdtext                | cmds__cmdtext_pkey                |        0 |            0 |             0
 16494 |      16505 | public     | tcc__ut1                     | tcc__ut1_pkey                     |        0 |            0 |             0
 16508 |      16514 | public     | mcp__userid                  | mcp__userid_pkey                  |        0 |            0 |             0
 16524 |      16527 | public     | tcc__usernum                 | tcc__usernum_pkey                 |        0 |            0 |             0
 16568 |      16571 | public     | tcc__instpos                 | tcc__instpos_pkey                 |        0 |            0 |             0
 16573 |      16579 | public     | tcc__gcview                  | tcc__gcview_pkey                  |        0 |            0 |             0
 16591 |      16606 | public     | tcc__facil                   | tcc__facil_pkey                   |        0 |            0 |             0
 16475 |      16491 | public     | tcc__etext                   | tcc__etext_pkey                   |        0 |            0 |             0
 16542 |      16545 | public     | tcc__scalefac                | tcc__scalefac_pkey                |        0 |            0 |             0
 16562 |      16581 | public     | tcc__inst                    | tcc__inst_pkey                    |        0 |            0 |             0
 16588 |      16605 | public     | tcc__vmsmsg                  | tcc__vmsmsg_pkey                  |        0 |            0 |             0
 16480 |      16486 | public     | tcc__tai                     | tcc__tai_pkey                     |        0 |            0 |             0
 16493 |      16499 | public     | tcc__lst                     | tcc__lst_pkey                     |        0 |            0 |             0
 16529 |      16532 | public     | tcc__useradded               | tcc__useradded_pkey               |        0 |            0 |             0
 16547 |      16550 | public     | tcc__secfocus                | tcc__secfocus_pkey                |        0 |            0 |             0
 16557 |      16560 | public     | tcc__started                 | tcc__started_pkey                 |        0 |            0 |             0
 16615 |      16618 | public     | tcc__objmag                  | tcc__objmag_pkey                  |        0 |            0 |             0
 16620 |      16623 | public     | tcc__objsys                  | tcc__objsys_pkey                  |        0 |            0 |             0
 16625 |      16628 | public     | tcc__objpos                  | tcc__objpos_pkey                  |        0 |            0 |             0
 16630 |      16633 | public     | tcc__objzpmpos               | tcc__objzpmpos_pkey               |        0 |            0 |             0
 16609 |      16635 | public     | tcc__objname                 | tcc__objname_pkey                 |        0 |            0 |             0
 16637 |      16643 | public     | tcc__objoff                  | tcc__objoff_pkey                  |        0 |            0 |             0
 16640 |      16644 | public     | tcc__boresight               | tcc__boresight_pkey               |        0 |            0 |             0
 16646 |      16649 | public     | tcc__objarcoff               | tcc__objarcoff_pkey               |        0 |            0 |             0
 16652 |      16655 | public     | tcc__guideoff                | tcc__guideoff_pkey                |        0 |            0 |             0
 16663 |      16666 | public     | tcc__currarcoff              | tcc__currarcoff_pkey              |        0 |            0 |             0
 16671 |      16674 | public     | tcc__rotpos                  | tcc__rotpos_pkey                  |        0 |            0 |             0
 16668 |      16676 | public     | tcc__objnetpos               | tcc__objnetpos_pkey               |        0 |            0 |             0
 16660 |      16678 | public     | tcc__rottype                 | tcc__rottype_pkey                 |        0 |            0 |             0
 16657 |      16680 | public     | tcc__caliboff                | tcc__caliboff_pkey                |        0 |            0 |             0
 16682 |      16688 | public     | tcc__objinstang              | tcc__objinstang_pkey              |        0 |            0 |             0
 16683 |      16690 | public     | tcc__spiderinstang           | tcc__spiderinstang_pkey           |        0 |            0 |             0
 16692 |      16695 | public     | mcp__controllamps            | mcp__controllamps_pkey            |        0 |            0 |             0
 16700 |      16711 | public     | mcp__semaphoreowner          | mcp__semaphoreowner_pkey          |        0 |            0 |             0
 16723 |      16739 | public     | tcc__moveitems               | tcc__moveitems_pkey               |        0 |            0 |             0
 16697 |      16709 | public     | mcp__commandin               | mcp__commandin_pkey               |        0 |            0 |             0
 16741 |      16744 | public     | tcc__primconstrms            | tcc__primconstrms_pkey            |        0 |            0 |             0
 16713 |      16716 | public     | tcc__axiscmdstate            | tcc__axiscmdstate_pkey            |        0 |            0 |             0
 16734 |      16737 | public     | tcc__slewbeg                 | tcc__slewbeg_pkey                 |        0 |            0 |             0
 16754 |      16759 | public     | tcc__secconstrms             | tcc__secconstrms_pkey             |        0 |            0 |             0
 16718 |      16721 | public     | tcc__slewadvtime             | tcc__slewadvtime_pkey             |        0 |            0 |             0
 16729 |      16732 | public     | tcc__slewduration            | tcc__slewduration_pkey            |        0 |            0 |             0
 16751 |      16757 | public     | tcc__secdesorient            | tcc__secdesorient_pkey            |        0 |            0 |             0
 16746 |      16749 | public     | tcc__primdesorient           | tcc__primdesorient_pkey           |        0 |            0 |             0
 16767 |      16770 | public     | tcc__tccpos                  | tcc__tccpos_pkey                  |        0 |            0 |             0
 16772 |      16775 | public     | tcc__axepos                  | tcc__axepos_pkey                  |        0 |            0 |             0
 16777 |      16783 | public     | tcc__badazstatus             | tcc__badazstatus_pkey             |        0 |            0 |             0
 16780 |      16785 | public     | tcc__azstat                  | tcc__azstat_pkey                  |        0 |            0 |             0
 16761 |      16787 | public     | tcc__tccstatus               | tcc__tccstatus_pkey               |        0 |            0 |             0
 16795 |      16801 | public     | tcc__badaltstatus            | tcc__badaltstatus_pkey            |        0 |            0 |             0
 16792 |      16803 | public     | tcc__altstat                 | tcc__altstat_pkey                 |        0 |            0 |             0
 16798 |      16805 | public     | tcc__altdtime                | tcc__altdtime_pkey                |        0 |            0 |             0
 16807 |      16810 | public     | tcc__rotstat                 | tcc__rotstat_pkey                 |        0 |            0 |             0
 16789 |      16812 | public     | tcc__azdtime                 | tcc__azdtime_pkey                 |        0 |            0 |             0
 16814 |      16818 | public     | tcc__badrotstatus            | tcc__badrotstatus_pkey            |        0 |            0 |             0
 16817 |      16825 | public     | tcc__rotid                   | tcc__rotid_pkey                   |        0 |            0 |             0
 16820 |      16826 | public     | tcc__rotdtime                | tcc__rotdtime_pkey                |        0 |            0 |             0
 16829 |      16832 | public     | tcc__trackadvtime            | tcc__trackadvtime_pkey            |        0 |            0 |             0
 16834 |      16840 | public     | tcc__cmddtime                | tcc__cmddtime_pkey                |        0 |            0 |             0
 16835 |      16842 | public     | tcc__primcmdmount            | tcc__primcmdmount_pkey            |        0 |            0 |             0
 16844 |      16847 | public     | tcc__primactmount            | tcc__primactmount_pkey            |        0 |            0 |             0
 16849 |      16852 | public     | tcc__primorient              | tcc__primorient_pkey              |        0 |            0 |             0
 16854 |      16857 | public     | tcc__seccmdmount             | tcc__seccmdmount_pkey             |        0 |            0 |             0
 16859 |      16862 | public     | tcc__secactmount             | tcc__secactmount_pkey             |        0 |            0 |             0
 16864 |      16867 | public     | tcc__secorient               | tcc__secorient_pkey               |        0 |            0 |             0
 16875 |      16878 | public     | tcc__iimscale                | tcc__iimscale_pkey                |        0 |            0 |             0
 16880 |      16883 | public     | tcc__iimctr                  | tcc__iimctr_pkey                  |        0 |            0 |             0
 16886 |      16891 | public     | tcc__gimscale                | tcc__gimscale_pkey                |        0 |            0 |             0
 16885 |      16892 | public     | tcc__iimlim                  | tcc__iimlim_pkey                  |        0 |            0 |             0
 16897 |      16903 | public     | tcc__gimctr                  | tcc__gimctr_pkey                  |        0 |            0 |             0
 16900 |      16904 | public     | tcc__gimlim                  | tcc__gimlim_pkey                  |        0 |            0 |             0
 16907 |      16910 | public     | tcc__rotlim                  | tcc__rotlim_pkey                  |        0 |            0 |             0
 16912 |      16915 | public     | tcc__slewend                 | tcc__slewend_pkey                 |        0 |            0 |             0
 16917 |      16923 | public     | tcc__gcamcmd                 | tcc__gcamcmd_pkey                 |        0 |            0 |             0
 16925 |      16928 | public     | mcp__fflamp                  | mcp__fflamp_pkey                  |        0 |            0 |             0
 16938 |      16942 | public     | mcp__nelampcommandedon       | mcp__nelampcommandedon_pkey       |        0 |            0 |             0
 16935 |      16939 | public     | mcp__nelamp                  | mcp__nelamp_pkey                  |        0 |            0 |             0
 16950 |      16953 | public     | mcp__hgcdlampcommandedon     | mcp__hgcdlampcommandedon_pkey     |        0 |            0 |             0
 16961 |      16966 | public     | tcc__sectrusstemp            | tcc__sectrusstemp_pkey            |        0 |            0 |             0
 16955 |      16968 | public     | tcc__airtemp                 | tcc__airtemp_pkey                 |        0 |            0 |             0
 16970 |      16973 | public     | tcc__secf_bftemp             | tcc__secf_bftemp_pkey             |        0 |            0 |             0
 16975 |      16978 | public     | tcc__pressure                | tcc__pressure_pkey                |        0 |            0 |             0
 16980 |      16989 | public     | tcc__humidity                | tcc__humidity_pkey                |        0 |            0 |             0
 16986 |      16990 | public     | tcc__tlapse                  | tcc__tlapse_pkey                  |        0 |            0 |             0
 16983 |      16993 | public     | tcc__winddir                 | tcc__winddir_pkey                 |        0 |            0 |             0
 17000 |      17003 | public     | tcc__timestamp               | tcc__timestamp_pkey               |        0 |            0 |             0
 17005 |      17008 | public     | tcc__primdesorientage        | tcc__primdesorientage_pkey        |        0 |            0 |             0
 17010 |      17013 | public     | tcc__primstatus              | tcc__primstatus_pkey              |        0 |            0 |             0
 16869 |      16894 | public     | tcc__ipconfig                | tcc__ipconfig_pkey                |        0 |            0 |             0
 16930 |      16933 | public     | mcp__fflampcommandedon       | mcp__fflampcommandedon_pkey       |        0 |            0 |             0
 16945 |      16948 | public     | mcp__hgcdlamp                | mcp__hgcdlamp_pkey                |        0 |            0 |             0
 16958 |      16964 | public     | tcc__primf_bftemp            | tcc__primf_bftemp_pkey            |        0 |            0 |             0
 16995 |      16998 | public     | tcc__windspeed               | tcc__windspeed_pkey               |        0 |            0 |             0
 17021 |      17024 | public     | tcc__secdesorientage         | tcc__secdesorientage_pkey         |        0 |            0 |             0
 17015 |      17026 | public     | tcc__received                | tcc__received_pkey                |        0 |            0 |             0
 17033 |      17039 | public     | tcc__axiserrcode             | tcc__axiserrcode_pkey             |        0 |            0 |             0
 17028 |      17031 | public     | tcc__secstatus               | tcc__secstatus_pkey               |        0 |            0 |             0
 17041 |      17047 | public     | tcc__axisinit                | tcc__axisinit_pkey                |        0 |            0 |             0
 17059 |      17065 | public     | tcc__axisstop                | tcc__axisstop_pkey                |        0 |            0 |             0
 17049 |      17052 | public     | tcc__userdeleted             | tcc__userdeleted_pkey             |        0 |            0 |             0
 17054 |      17057 | public     | mcp__rotbadfiducialdelta     | mcp__rotbadfiducialdelta_pkey     |        0 |            0 |             0
 17067 |      17070 | public     | mcp__ffsstatus               | mcp__ffsstatus_pkey               |        0 |            0 |             0
 17072 |      17075 | public     | mcp__ffscommandedon          | mcp__ffscommandedon_pkey          |        0 |            0 |             0
 17077 |      17080 | public     | mcp__needsemaphore           | mcp__needsemaphore_pkey           |        0 |            0 |             0
 17082 |      17088 | public     | tcc__axisnotrack             | tcc__axisnotrack_pkey             |        0 |            0 |             0
 17090 |      17096 | public     | tcc__axisnoslew              | tcc__axisnoslew_pkey              |        0 |            0 |             0
 17098 |      17101 | public     | tcc__schmovetimes            | tcc__schmovetimes_pkey            |        0 |            0 |             0
 17103 |      17106 | public     | tcc__objpm                   | tcc__objpm_pkey                   |        0 |            0 |             0
 17108 |      17111 | public     | tcc__rotwrappref             | tcc__rotwrappref_pkey             |        0 |            0 |             0
 17113 |      17116 | public     | tcc__ptrefpos                | tcc__ptrefpos_pkey                |        0 |            0 |             0
 17118 |      17121 | public     | tcc__azaltdist               | tcc__azaltdist_pkey               |        0 |            0 |             0
 17123 |      17126 | public     | tcc__moved                   | tcc__moved_pkey                   |        0 |            0 |             0
 17128 |      17131 | public     | tcc__telmount                | tcc__telmount_pkey                |        0 |            0 |             0
 17133 |      17136 | public     | tcc__rotmount                | tcc__rotmount_pkey                |        0 |            0 |             0
 17138 |      17141 | public     | tcc__msgslost                | tcc__msgslost_pkey                |        0 |            0 |             0
 17143 |      17146 | public     | tcc__baduser                 | tcc__baduser_pkey                 |        0 |            0 |             0
 17148 |      17151 | public     | tcc__userinfo                | tcc__userinfo_pkey                |        0 |            0 |             0
 17153 |      17156 | public     | mcp__ffsmoved                | mcp__ffsmoved_pkey                |        0 |            0 |             0
 17158 |      17161 | public     | mcp__ffsopenfailed           | mcp__ffsopenfailed_pkey           |        0 |            0 |             0
 17163 |      17169 | public     | tcc__expected                | tcc__expected_pkey                |        0 |            0 |             0
 17171 |      17174 | public     | tcc__slewsuperseded          | tcc__slewsuperseded_pkey          |        0 |            0 |             0
 17189 |      17192 | public     | tcc__warnrotstatus           | tcc__warnrotstatus_pkey           |        0 |            0 |             0
 17223 |      17242 | public     | perms__authlist              | perms__authlist_pkey              |        0 |            0 |             0
 17259 |      17262 | public     | mcp__saddleismounted         | mcp__saddleismounted_pkey         |        0 |            0 |             0
 17274 |      17296 | public     | mcp__altfiducialversion      | mcp__altfiducialversion_pkey      |        0 |            0 |             0
 17308 |      17311 | public     | mcp__minencodermismatch      | mcp__minencodermismatch_pkey      |        0 |            0 |             0
 17176 |      17182 | public     | tcc__rejectedaxiserrcode     | tcc__rejectedaxiserrcode_pkey     |        0 |            0 |             0
 17221 |      17238 | public     | perms__actors                | perms__actors_pkey                |        0 |            0 |             0
 17244 |      17247 | public     | mcp__ffsselected             | mcp__ffsselected_pkey             |        0 |            0 |             0
 17280 |      17292 | public     | mcp__azfiducialversion       | mcp__azfiducialversion_pkey       |        0 |            0 |             0
 17313 |      17316 | public     | mcp__plcversion              | mcp__plcversion_pkey              |        0 |            0 |             0
 17326 |      17334 | public     | mcp__cwpositions             | mcp__cwpositions_pkey             |        0 |            0 |             0
 17184 |      17187 | public     | tcc__warnaltstatus           | tcc__warnaltstatus_pkey           |        0 |            0 |             0
 17194 |      17197 | public     | mcp__rotmaxaccrequested      | mcp__rotmaxaccrequested_pkey      |        0 |            0 |             0
 17199 |      17202 | public     | tcc__warnazstatus            | tcc__warnazstatus_pkey            |        0 |            0 |             0
 17220 |      17240 | public     | perms__programs              | perms__programs_pkey              |        0 |            0 |             0
 17252 |      17257 | public     | mcp__whtlampcommandedon      | mcp__whtlampcommandedon_pkey      |        0 |            0 |             0
 17269 |      17272 | public     | mcp__instrumentnum           | mcp__instrumentnum_pkey           |        0 |            0 |             0
 17299 |      17305 | public     | mcp__msonmaxcorrection       | mcp__msonmaxcorrection_pkey       |        0 |            0 |             0
 17204 |      17210 | public     | hub__users                   | hub__users_pkey                   |        0 |            0 |             0
 17264 |      17267 | public     | mcp__instrumentnumconsistent | mcp__instrumentnumconsistent_pkey |        0 |            0 |             0
 17298 |      17304 | public     | mcp__goodfiducialversions    | mcp__goodfiducialversions_pkey    |        0 |            0 |             0
 17327 |      17332 | public     | mcp__cwstatus                | mcp__cwstatus_pkey                |        0 |            0 |             0
 17212 |      17218 | public     | hub__user                    | hub__user_pkey                    |        0 |            0 |             0
 17249 |      17255 | public     | mcp__uvlampcommandedon       | mcp__uvlampcommandedon_pkey       |        0 |            0 |             0
 17283 |      17294 | public     | mcp__rotfiducialversion      | mcp__rotfiducialversion_pkey      |        0 |            0 |             0
 17318 |      17324 | public     | mcp__mcpversion              | mcp__mcpversion_pkey              |        0 |            0 |             0
 17336 |      17339 | public     | mcp__lampscommanded          | mcp__lampscommanded_pkey          |        0 |            0 |             0
 17341 |      17344 | public     | mcp__azmson                  | mcp__azmson_pkey                  |        0 |            0 |             0
 17346 |      17349 | public     | mcp__altmson                 | mcp__altmson_pkey                 |        0 |            0 |             0
 17351 |      17354 | public     | mcp__rotmson                 | mcp__rotmson_pkey                 |        0 |            0 |             0
 17356 |      17359 | public     | mcp__ffscommanded            | mcp__ffscommanded_pkey            |        0 |            0 |             0
 17361 |      17364 | public     | mcp__azmaxaccrequested       | mcp__azmaxaccrequested_pkey       |        0 |            0 |             0
 17366 |      17372 | public     | mcp__text                    | mcp__text_pkey                    |        0 |            0 |             0
 17791 |      17794 | public     | tcc__exptime                 | tcc__exptime_pkey                 |        0 |            0 |             0
 17796 |      17799 | public     | tcc__superseded              | tcc__superseded_pkey              |        0 |            0 |             0
 17801 |      17807 | public     | hub__text                    | hub__text_pkey                    |        0 |            0 |             0
 17809 |      17815 | public     | hub__actors                  | hub__actors_pkey                  |        0 |            0 |             0
 18299 |      18302 | public     | tcc__badazdtime              | tcc__badazdtime_pkey              |        0 |            0 |             0
(179 rows)

Look at postgres disk usage statistics following these instructions:

test=# select sum(relpages) from pg_class where relowner=16385;
 sum
--------
 392029
(1 row)

Each page is 8k so the total disk usage here is 3062.7Mb (what about tables with zero pages??) List tables sorted by size:

test=# select relname,relpages,relfilenode from pg_class where relowner=16385 order by relpages desc;
 relname                           | relpages | relfilenode
-----------------------------------+----------+-------------
 reply_raw                         |   105251 |       16386
 reply_hdr                         |    56411 |       16389
 reply_hdr_actor_id__int4          |    26617 |       16402
 reply_hdr_pkey                    |    20983 |       16400
 reply_raw_tai__flt8               |    20975 |       16403
 reply_raw_pkey                    |    20975 |       16398
 mcp__commandin                    |    13997 |       16697
 cmds__cmdtext                     |     6448 |       16472
 cmds__cmdrmid                     |     6356 |       16463
 cmds__cmdtime                     |     6349 |       16444
 cmds__newcmd                      |     6349 |       16441
 cmds__cmdactor                    |     6346 |       16449
 cmds__cmdr                        |     6341 |       16453
 cmds__cmddone                     |     4538 |       16436
 mcp__commandin_pkey               |     4464 |       16709
 cmds__cmdtext_pkey                |     3224 |       16489
 cmds__cmdtime_pkey                |     3174 |       16452
 cmds__newcmd_pkey                 |     3174 |       16447
 cmds__cmdr_pkey                   |     3173 |       16468
 cmds__cmdrmid_pkey                |     3173 |       16466
 cmds__cmdactor_pkey               |     3173 |       16470
 mcp__command                      |     3095 |       16516
 cmds__cmddone_pkey                |     2259 |       16439
 tcc__cmd                          |     2169 |       16534
 tcc__trackadvtime                 |     2127 |       16829
 tcc__axepos                       |     1719 |       16772
 tcc__tccpos                       |     1707 |       16767
 mcp__command_pkey                 |     1473 |       16522
 tcc__tccstatus                    |     1457 |       16761
 tcc__text                         |     1421 |       16420
 tcc__spiderinstang                |     1245 |       16683
 tcc__trackadvtime_pkey            |     1038 |       16832
 tcc__azstat                       |      949 |       16780
 tcc__altstat                      |      924 |       16792
 tcc__rotstat                      |      922 |       16807
 tcc__objoff                       |      870 |       16637
 tcc__cmd_pkey                     |      823 |       16540
 tcc__azdtime                      |      735 |       16789
 tcc__caliboff                     |      728 |       16657
 tcc__guideoff                     |      715 |       16652
 tcc__axepos_pkey                  |      714 |       16775
 tcc__tccpos_pkey                  |      707 |       16770
 tcc__tccstatus_pkey               |      703 |       16787
 tcc__altdtime                     |      696 |       16798
 tcc__rotid                        |      693 |       16817
 tcc__rotdtime                     |      692 |       16820
 tcc__rotpos                       |      669 |       16671
 tcc__modu                         |      644 |       16426
 tcc__objpos                       |      622 |       16625
 tcc__objzpmpos                    |      619 |       16630
 tcc__objnetpos                    |      611 |       16668
 tcc__badrotstatus                 |      582 |       16814
 tcc__objarcoff                    |      582 |       16646
 tcc__badazstatus                  |      576 |       16777
 tcc__boresight                    |      573 |       16640
 tcc__rottype                      |      568 |       16660
 tcc__badaltstatus                 |      563 |       16795
 tcc__gcamcmd                      |      546 |       16917
 tcc__spiderinstang_pkey           |      513 |       16690
 tcc__moveitems                    |      511 |       16723
 tcc__started                      |      494 |       16557
 tcc__objinstang                   |      461 |       16682
 tcc__text_pkey                    |      458 |       16432
 tcc__objpm                        |      442 |       17103
 tcc__objname                      |      423 |       16609
 tcc__cmddtime                     |      423 |       16834
 tcc__objsys                       |      421 |       16620
 tcc__objmag                       |      421 |       16615
 tcc__azdtime_pkey                 |      347 |       16812
 tcc__moved                        |      338 |       17123
 tcc__secorient                    |      337 |       16864
 tcc__primorient                   |      335 |       16849
 tcc__primdesorient                |      334 |       16746
 tcc__badazstatus_pkey             |      333 |       16783
 tcc__secdesorient                 |      329 |       16751
 tcc__azstat_pkey                  |      329 |       16785
 tcc__badrotstatus_pkey            |      328 |       16818
 tcc__altstat_pkey                 |      326 |       16803
 tcc__altdtime_pkey                |      326 |       16805
 tcc__rotdtime_pkey                |      325 |       16826
 tcc__rotstat_pkey                 |      325 |       16810
 tcc__rotid_pkey                   |      325 |       16825
 tcc__badaltstatus_pkey            |      319 |       16801
 tcc__telmount                     |      298 |       17128
 tcc__started_pkey                 |      277 |       16560
 tcc__secactmount                  |      273 |       16859
 tcc__secconstrms                  |      272 |       16754
 tcc__primcmdmount                 |      272 |       16835
 tcc__primconstrms                 |      269 |       16741
 tcc__objoff_pkey                  |      267 |       16643
 tcc__rotpos_pkey                  |      267 |       16674
 tcc__primactmount                 |      267 |       16844
 tcc__rottype_pkey                 |      263 |       16678
 tcc__seccmdmount                  |      248 |       16854
 tcc__ut1                          |      245 |       16494
 tcc__utc_tai                      |      239 |       16501
 tcc__lst                          |      236 |       16493
 tcc__tai                          |      235 |       16480
 tcc__modu_pkey                    |      231 |       16434
 tcc__rotmount                     |      229 |       17133
 tcc__windspeed                    |      224 |       16995
 tcc__winddir                      |      220 |       16983
 tcc__sectrusstemp                 |      220 |       16961
 tcc__tlapse                       |      218 |       16986
 tcc__timestamp                    |      215 |       17000
 tcc__secfocus                     |      214 |       16547
 tcc__airtemp                      |      214 |       16955
 tcc__secf_bftemp                  |      212 |       16970
 tcc__humidity                     |      211 |       16980
 tcc__pressure                     |      210 |       16975
 tcc__primf_bftemp                 |      208 |       16958
 tcc__gcfocus                      |      203 |       16552
 tcc__objname_pkey                 |      188 |       16635
 tcc__objsys_pkey                  |      187 |       16623
 tcc__objpos_pkey                  |      187 |       16628
 tcc__objmag_pkey                  |      187 |       16618
 tcc__moveitems_pkey               |      187 |       16739
 tcc__objnetpos_pkey               |      187 |       16676
 tcc__objzpmpos_pkey               |      185 |       16633
 tcc__objinstang_pkey              |      182 |       16688
 tcc__caliboff_pkey                |      179 |       16680
 tcc__guideoff_pkey                |      177 |       16655
 tcc__boresight_pkey               |      177 |       16644
 tcc__objarcoff_pkey               |      177 |       16649
 tcc__cmddtime_pkey                |      172 |       16840
 tcc__moved_pkey                   |      170 |       17126
 tcc__objpm_pkey                   |      170 |       17106
 tcc__ut1_pkey                     |       98 |       16505
 tcc__gcamcmd_pkey                 |       98 |       16923
 tcc__utc_tai_pkey                 |       97 |       16504
 tcc__secorient_pkey               |       95 |       16867
 tcc__secactmount_pkey             |       95 |       16862
 tcc__secconstrms_pkey             |       94 |       16759
 tcc__primactmount_pkey            |       94 |       16847
 tcc__secdesorient_pkey            |       94 |       16757
 tcc__primorient_pkey              |       94 |       16852
 tcc__primconstrms_pkey            |       94 |       16744
 tcc__primcmdmount_pkey            |       94 |       16842
 tcc__lst_pkey                     |       93 |       16499
 tcc__primdesorient_pkey           |       93 |       16749
 tcc__secfocus_pkey                |       92 |       16550
 tcc__winddir_pkey                 |       92 |       16993
 tcc__windspeed_pkey               |       92 |       16998
 tcc__timestamp_pkey               |       92 |       17003
 tcc__tlapse_pkey                  |       92 |       16990
 tcc__tai_pkey                     |       91 |       16486
 tcc__sectrusstemp_pkey            |       91 |       16966
 tcc__airtemp_pkey                 |       91 |       16968
 tcc__primf_bftemp_pkey            |       88 |       16964
 tcc__pressure_pkey                |       88 |       16978
 tcc__humidity_pkey                |       88 |       16989
 tcc__secf_bftemp_pkey             |       88 |       16973
 tcc__seccmdmount_pkey             |       87 |       16857
 tcc__rotmount_pkey                |       86 |       17136
 tcc__telmount_pkey                |       86 |       17131
 tcc__gcfocus_pkey                 |       86 |       16555
 tcc__axiscmdstate                 |       75 |       16713
 tcc__currarcoff                   |       62 |       16663
 tcc__axiserrcode                  |       60 |       17033
 tcc__etext                        |       40 |       16475
 tcc__usernum                      |       40 |       16524
 tcc__slewadvtime                  |       31 |       16718
 tcc__slewbeg                      |       30 |       16734
 tcc__slewduration                 |       30 |       16729
 tcc__received                     |       28 |       17015
 tcc__scalefac                     |       25 |       16542
 mcp__userid                       |       25 |       16508
 tcc__slewend                      |       19 |       16912
 mcp__controllamps                 |       18 |       16692
 tcc__axiscmdstate_pkey            |       17 |       16716
 tcc__axisinit                     |       14 |       17041
 mcp__semaphoreowner               |       12 |       16700
 mcp__fflamp                       |       11 |       16925
 mcp__fflampcommandedon            |       11 |       16930
 mcp__ffsstatus                    |       11 |       17067
 mcp__ffscommandedon               |       10 |       17072
 tcc__currarcoff_pkey              |       10 |       16666
 tcc__axiserrcode_pkey             |        9 |       17039
 tcc__msgslost                     |        9 |       17138
 tcc__usernum_pkey                 |        8 |       16527
 tcc__secstatus                    |        6 |       17028
 tcc__axisstop                     |        6 |       17059
 mcp__rotbadfiducialdelta          |        6 |       17054
 mcp__controllamps_pkey            |        6 |       16695
 tcc__scalefac_pkey                |        6 |       16545
 tcc__primdesorientage             |        6 |       17005
 tcc__secdesorientage              |        6 |       17021
 tcc__rotwrappref                  |        6 |       17108
 tcc__primstatus                   |        6 |       17010
 tcc__ptrefpos                     |        6 |       17113
 tcc__slewbeg_pkey                 |        6 |       16737
 tcc__etext_pkey                   |        6 |       16491
 tcc__slewend_pkey                 |        6 |       16915
 tcc__slewadvtime_pkey             |        6 |       16721
 tcc__slewduration_pkey            |        6 |       16732
 tcc__received_pkey                |        5 |       17026
 mcp__userid_pkey                  |        5 |       16514
 tcc__useradded                    |        5 |       16529
 tcc__azaltdist                    |        5 |       17118
 mcp__fflampcommandedon_pkey       |        4 |       16933
 mcp__ffscommandedon_pkey          |        4 |       17075
 tcc__userdeleted                  |        4 |       17049
 mcp__fflamp_pkey                  |        4 |       16928
 mcp__semaphoreowner_pkey          |        4 |       16711
 mcp__ffsstatus_pkey               |        4 |       17070
 tcc__msgslost_pkey                |        4 |       17141
 mcp__rotbadfiducialdelta_pkey     |        4 |       17057
 tcc__facil                        |        4 |       16591
 tcc__axisinit_pkey                |        4 |       17047
 tcc__vmsmsg                       |        3 |       16588
 tcc__primstatus_pkey              |        2 |       17013
 tcc__rotlim_pkey                  |        2 |       16910
 tcc__ipconfig_pkey                |        2 |       16894
 mcp__needsemaphore_pkey           |        2 |       17080
 tcc__slewsuperseded_pkey          |        2 |       17174
 mcp__hgcdlamp_pkey                |        2 |       16948
 tcc__failed_pkey                  |        2 |       16603
 tcc__expected_pkey                |        2 |       17169
 tcc__gcnfilt_pkey                 |        2 |       16586
 tcc__userinfo_pkey                |        2 |       17151
 tcc__warnaltstatus_pkey           |        2 |       17187
 tcc__secdesorientage_pkey         |        2 |       17024
 tcc__inst_pkey                    |        2 |       16581
 tcc__userdeleted_pkey             |        2 |       17052
 tcc__iimscale_pkey                |        2 |       16878
 tcc__secstatus_pkey               |        2 |       17031
 tcc__axisstop_pkey                |        2 |       17065
 tcc__rejectedaxiserrcode_pkey     |        2 |       17182
 tcc__iimctr_pkey                  |        2 |       16883
 mcp__nelampcommandedon_pkey       |        2 |       16942
 mcp__nelamp_pkey                  |        2 |       16939
 tcc__warnazstatus_pkey            |        2 |       17202
 mcp__hgcdlampcommandedon_pkey     |        2 |       16953
 tcc__baduser_pkey                 |        2 |       17146
 tcc__warnrotstatus_pkey           |        2 |       17192
 tcc__facil_pkey                   |        2 |       16606
 tcc__gimscale_pkey                |        2 |       16891
 tcc__gcview_pkey                  |        2 |       16579
 tcc__azaltdist_pkey               |        2 |       17121
 tcc__iimlim_pkey                  |        2 |       16892
 tcc__useradded_pkey               |        2 |       16532
 tcc__ptrefpos_pkey                |        2 |       17116
 tcc__instpos_pkey                 |        2 |       16571
 tcc__vmsmsg_pkey                  |        2 |       16605
 tcc__gimctr_pkey                  |        2 |       16903
 tcc__rotwrappref_pkey             |        2 |       17111
 tcc__gimlim_pkey                  |        2 |       16904
 tcc__axisnoslew_pkey              |        2 |       17096
 tcc__primdesorientage_pkey        |        2 |       17008
 pg_toast_16761_index              |        1 |       16766
 pg_toast_16723_index              |        1 |       16728
 pg_toast_16700_index              |        1 |       16708
 pg_toast_16697_index              |        1 |       16705
 pg_toast_16609_index              |        1 |       16614
 pg_toast_16869_index              |        1 |       16874
 tcc__iimscale                     |        1 |       16875
 tcc__iimctr                       |        1 |       16880
 tcc__iimlim                       |        1 |       16885
 tcc__gimscale                     |        1 |       16886
 tcc__gimctr                       |        1 |       16897
 tcc__gimlim                       |        1 |       16900
 tcc__rotlim                       |        1 |       16907
 pg_toast_16588_index              |        1 |       16599
 pg_toast_16917_index              |        1 |       16922
 tcc__inst                         |        1 |       16562
 pg_toast_16562_index              |        1 |       16567
 mcp__nelamp                       |        1 |       16935
 mcp__nelampcommandedon            |        1 |       16938
 mcp__hgcdlampcommandedon          |        1 |       16950
 pg_toast_16475_index              |        1 |       16485
 pg_toast_16591_index              |        1 |       16597
 tcc__gcview                       |        1 |       16573
 pg_toast_16573_index              |        1 |       16578
 tcc__instpos                      |        1 |       16568
 pg_toast_16508_index              |        1 |       16513
 pg_toast_16472_index              |        1 |       16488
 pg_toast_16449_index              |        1 |       16462
 tcc__ipconfig                     |        1 |       16869
 mcp__hgcdlamp                     |        1 |       16945
 tcc__failed                       |        1 |       16600
 tcc__gcnfilt                      |        1 |       16583
 pg_toast_16534_index              |        1 |       16539
 pg_toast_17015_index              |        1 |       17020
 pg_toast_17033_index              |        1 |       17038
 pg_toast_16516_index              |        1 |       16521
 pg_toast_16453_index              |        1 |       16461
 pg_toast_17041_index              |        1 |       17046
 pg_toast_17059_index              |        1 |       17064
 pg_toast_16426_index              |        1 |       16431
 pg_toast_16420_index              |        1 |       16425
 mcp__needsemaphore                |        1 |       17077
 pg_toast_17082_index              |        1 |       17087
 tcc__axisnotrack_pkey             |        1 |       17088
 pg_toast_17090_index              |        1 |       17095
 tcc__axisnoslew                   |        1 |       17090
 tcc__schmovetimes_pkey            |        1 |       17101
 actors_pkey                       |        1 |       16418
 hub__commanders_pkey              |        1 |       16416
 tcc__badazdtime_pkey              |        1 |       18302
 pg_toast_16410_index              |        1 |       16415
 pg_toast_16404_index              |        1 |       16409
 pg_toast_16386_index              |        1 |       16397
 pg_toast_16389_index              |        1 |       16396
 tcc__baduser                      |        1 |       17143
 tcc__userinfo                     |        1 |       17148
 mcp__ffsmoved_pkey                |        1 |       17156
 mcp__ffsopenfailed_pkey           |        1 |       17161
 pg_toast_17163_index              |        1 |       17168
 tcc__expected                     |        1 |       17163
 tcc__slewsuperseded               |        1 |       17171
 tcc__warnrotstatus                |        1 |       17189
 pg_toast_17223_index              |        1 |       17237
 perms__authlist_pkey              |        1 |       17242
 mcp__saddleismounted_pkey         |        1 |       17262
 pg_toast_17274_index              |        1 |       17279
 mcp__altfiducialversion_pkey      |        1 |       17296
 mcp__minencodermismatch_pkey      |        1 |       17311
 pg_toast_17176_index              |        1 |       17181
 tcc__rejectedaxiserrcode          |        1 |       17176
 pg_toast_17221_index              |        1 |       17231
 perms__actors_pkey                |        1 |       17238
 mcp__ffsselected_pkey             |        1 |       17247
 pg_toast_17280_index              |        1 |       17288
 mcp__azfiducialversion_pkey       |        1 |       17292
 mcp__plcversion_pkey              |        1 |       17316
 mcp__cwpositions_pkey             |        1 |       17334
 tcc__warnaltstatus                |        1 |       17184
 mcp__rotmaxaccrequested_pkey      |        1 |       17197
 tcc__warnazstatus                 |        1 |       17199
 pg_toast_17220_index              |        1 |       17234
 perms__programs_pkey              |        1 |       17240
 mcp__whtlampcommandedon_pkey      |        1 |       17257
 mcp__instrumentnum_pkey           |        1 |       17272
 mcp__msonmaxcorrection_pkey       |        1 |       17305
 pg_toast_17204_index              |        1 |       17209
 hub__users_pkey                   |        1 |       17210
 mcp__instrumentnumconsistent_pkey |        1 |       17267
 mcp__goodfiducialversions_pkey    |        1 |       17304
 mcp__cwstatus_pkey                |        1 |       17332
 pg_toast_17212_index              |        1 |       17217
 hub__user_pkey                    |        1 |       17218
 mcp__uvlampcommandedon_pkey       |        1 |       17255
 pg_toast_17283_index              |        1 |       17291
 mcp__rotfiducialversion_pkey      |        1 |       17294
 pg_toast_17318_index              |        1 |       17323
 mcp__mcpversion_pkey              |        1 |       17324
 mcp__lampscommanded_pkey          |        1 |       17339
 mcp__azmson_pkey                  |        1 |       17344
 mcp__altmson_pkey                 |        1 |       17349
 mcp__rotmson_pkey                 |        1 |       17354
 mcp__ffscommanded_pkey            |        1 |       17359
 mcp__azmaxaccrequested_pkey       |        1 |       17364
 pg_toast_17366_index              |        1 |       17371
 mcp__text_pkey                    |        1 |       17372
 tcc__exptime_pkey                 |        1 |       17794
 tcc__superseded_pkey              |        1 |       17799
 pg_toast_17801_index              |        1 |       17806
 hub__text_pkey                    |        1 |       17807
 pg_toast_17809_index              |        1 |       17814
 hub__actors_pkey                  |        1 |       17815
 pg_toast_17221                    |        0 |       17229
 mcp__azmaxaccrequested            |        0 |       17361
 perms__actors                     |        0 |       17221
 mcp__uvlampcommandedon            |        0 |       17249
 mcp__ffsselected                  |        0 |       17244
 hub__actors                       |        0 |       17809
 pg_toast_17280                    |        0 |       17284
 pg_toast_17283                    |        0 |       17289
 mcp__azfiducialversion            |        0 |       17280
 pg_toast_17366                    |        0 |       17369
 mcp__plcversion                   |        0 |       17313
 pg_toast_16453                    |        0 |       16459
 pg_toast_17059                    |        0 |       17062
 mcp__rotfiducialversion           |        0 |       17283
 mcp__cwpositions                  |        0 |       17326
 pg_toast_17041                    |        0 |       17044
 pg_toast_16516                    |        0 |       16519
 pg_toast_17033                    |        0 |       17036
 pg_toast_16426                    |        0 |       16429
 pg_toast_16389                    |        0 |       16392
 hub__text                         |        0 |       17801
 pg_toast_16534                    |        0 |       16537
 pg_toast_16420                    |        0 |       16423
 pg_toast_17015                    |        0 |       17018
 pg_toast_17318                    |        0 |       17321
 pg_toast_16449                    |        0 |       16456
 pg_toast_17082                    |        0 |       17085
 pg_toast_16700                    |        0 |       16706
 tcc__axisnotrack                  |        0 |       17082
 mcp__rotmaxaccrequested           |        0 |       17194
 pg_toast_17090                    |        0 |       17093
 mcp__text                         |        0 |       17366
 mcp__mcpversion                   |        0 |       17318
 pg_toast_16472                    |        0 |       16478
 tcc__schmovetimes                 |        0 |       17098
 pg_toast_16723                    |        0 |       16726
 pg_toast_17220                    |        0 |       17232
 actors                            |        0 |       16410
 pg_toast_16761                    |        0 |       16764
 pg_toast_16508                    |        0 |       16511
 hub__commanders                   |        0 |       16404
 pg_toast_16573                    |        0 |       16576
 perms__programs                   |        0 |       17220
 pg_toast_16591                    |        0 |       16595
 mcp__lampscommanded               |        0 |       17336
 pg_toast_16410                    |        0 |       16413
 mcp__whtlampcommandedon           |        0 |       17252
 pg_toast_16404                    |        0 |       16407
 tcc__exptime                      |        0 |       17791
 mcp__instrumentnum                |        0 |       17269
 pg_toast_16386                    |        0 |       16394
 pg_toast_16475                    |        0 |       16483
 mcp__azmson                       |        0 |       17341
 pg_toast_16562                    |        0 |       16565
 mcp__msonmaxcorrection            |        0 |       17299
 pg_toast_16917                    |        0 |       16920
 mcp__ffsmoved                     |        0 |       17153
 pg_toast_17809                    |        0 |       17812
 mcp__ffsopenfailed                |        0 |       17158
 pg_toast_17204                    |        0 |       17207
 pg_toast_17163                    |        0 |       17166
 mcp__altmson                      |        0 |       17346
 hub__users                        |        0 |       17204
 pg_toast_16588                    |        0 |       16592
 tcc__superseded                   |        0 |       17796
 pg_toast_16869                    |        0 |       16872
 mcp__instrumentnumconsistent      |        0 |       17264
 pg_toast_16609                    |        0 |       16612
 pg_toast_17223                    |        0 |       17235
 mcp__rotmson                      |        0 |       17351
 perms__authlist                   |        0 |       17223
 mcp__goodfiducialversions         |        0 |       17298
 mcp__saddleismounted              |        0 |       17259
 tcc__badazdtime                   |        0 |       18299
 pg_toast_17274                    |        0 |       17277
 mcp__cwstatus                     |        0 |       17327
 mcp__altfiducialversion           |        0 |       17274
 mcp__ffscommanded                 |        0 |       17356
 mcp__minencodermismatch           |        0 |       17308
 pg_toast_17212                    |        0 |       17215
 pg_toast_17176                    |        0 |       17179
 pg_toast_17801                    |        0 |       17804
 hub__user                         |        0 |       17212
 pg_toast_16697                    |        0 |       16703
(444 rows)

The filenode identifies the corresponding file under /var/lib/pgsql/data but I don’t have priv to see this directory. Write to Fritz asking him to add the priv for me.

The pg_toast_N entries are for wide columns and the pg_toast_N_index entries are for their indices (each one has one index?):

test=# select c.relname,c2.relname,c.relpages from pg_class as c,pg_class as c2 where c.relowner=16385 and c.oid=c2.reltoastrelid order by c.relpages desc;
 relname     |         relname          | relpages
----------------+--------------------------+----------
 pg_toast_16386 | reply_raw                |        0
 pg_toast_16389 | reply_hdr                |        0
 pg_toast_16404 | hub__commanders          |        0
 pg_toast_16410 | actors                   |        0
 pg_toast_16420 | tcc__text                |        0
 pg_toast_16426 | tcc__modu                |        0
 pg_toast_16453 | cmds__cmdr               |        0
 pg_toast_16516 | mcp__command             |        0
 pg_toast_16534 | tcc__cmd                 |        0
 pg_toast_16449 | cmds__cmdactor           |        0
 pg_toast_16472 | cmds__cmdtext            |        0
 pg_toast_16508 | mcp__userid              |        0
 pg_toast_16573 | tcc__gcview              |        0
 pg_toast_16591 | tcc__facil               |        0
 pg_toast_16475 | tcc__etext               |        0
 pg_toast_16562 | tcc__inst                |        0
 pg_toast_16588 | tcc__vmsmsg              |        0
 pg_toast_16609 | tcc__objname             |        0
 pg_toast_16700 | mcp__semaphoreowner      |        0
 pg_toast_16723 | tcc__moveitems           |        0
 pg_toast_16697 | mcp__commandin           |        0
 pg_toast_16761 | tcc__tccstatus           |        0
 pg_toast_16917 | tcc__gcamcmd             |        0
 pg_toast_16869 | tcc__ipconfig            |        0
 pg_toast_17015 | tcc__received            |        0
 pg_toast_17033 | tcc__axiserrcode         |        0
 pg_toast_17041 | tcc__axisinit            |        0
 pg_toast_17059 | tcc__axisstop            |        0
 pg_toast_17082 | tcc__axisnotrack         |        0
 pg_toast_17090 | tcc__axisnoslew          |        0
 pg_toast_17163 | tcc__expected            |        0
 pg_toast_17223 | perms__authlist          |        0
 pg_toast_17274 | mcp__altfiducialversion  |        0
 pg_toast_17176 | tcc__rejectedaxiserrcode |        0
 pg_toast_17221 | perms__actors            |        0
 pg_toast_17280 | mcp__azfiducialversion   |        0
 pg_toast_17220 | perms__programs          |        0
 pg_toast_17204 | hub__users               |        0
 pg_toast_17212 | hub__user                |        0
 pg_toast_17283 | mcp__rotfiducialversion  |        0
 pg_toast_17318 | mcp__mcpversion          |        0
 pg_toast_17366 | mcp__text                |        0
 pg_toast_17801 | hub__text                |        0
 pg_toast_17809 | hub__actors              |        0
(44 rows)

Here is the complete list of indices:

test=# select c.relname,c2.relname,c.relpages from pg_class as c,pg_class as c2,pg_index as i where c.relowner=16385 and c.oid=i.indexrelid and c2.oid=indrelid order by c.relpages desc;
 relname              |           relname            | relpages
-----------------------------------+------------------------------+----------
 reply_hdr_actor_id__int4          | reply_hdr                    |    26617
 reply_hdr_pkey                    | reply_hdr                    |    20983
 reply_raw_pkey                    | reply_raw                    |    20975
 reply_raw_tai__flt8               | reply_raw                    |    20975
 mcp__commandin_pkey               | mcp__commandin               |     4464
 cmds__cmdtext_pkey                | cmds__cmdtext                |     3224
 cmds__cmdtime_pkey                | cmds__cmdtime                |     3174
 cmds__newcmd_pkey                 | cmds__newcmd                 |     3174
 cmds__cmdrmid_pkey                | cmds__cmdrmid                |     3173
 cmds__cmdactor_pkey               | cmds__cmdactor               |     3173
 cmds__cmdr_pkey                   | cmds__cmdr                   |     3173
 cmds__cmddone_pkey                | cmds__cmddone                |     2259
 mcp__command_pkey                 | mcp__command                 |     1473
 tcc__trackadvtime_pkey            | tcc__trackadvtime            |     1038
 tcc__cmd_pkey                     | tcc__cmd                     |      823
 tcc__axepos_pkey                  | tcc__axepos                  |      714
 tcc__tccpos_pkey                  | tcc__tccpos                  |      707
 tcc__tccstatus_pkey               | tcc__tccstatus               |      703
 tcc__spiderinstang_pkey           | tcc__spiderinstang           |      513
 tcc__text_pkey                    | tcc__text                    |      458
 tcc__azdtime_pkey                 | tcc__azdtime                 |      347
 tcc__badazstatus_pkey             | tcc__badazstatus             |      333
 tcc__azstat_pkey                  | tcc__azstat                  |      329
 tcc__badrotstatus_pkey            | tcc__badrotstatus            |      328
 tcc__altstat_pkey                 | tcc__altstat                 |      326
 tcc__altdtime_pkey                | tcc__altdtime                |      326
 tcc__rotstat_pkey                 | tcc__rotstat                 |      325
 tcc__rotid_pkey                   | tcc__rotid                   |      325
 tcc__rotdtime_pkey                | tcc__rotdtime                |      325
 tcc__badaltstatus_pkey            | tcc__badaltstatus            |      319
 tcc__started_pkey                 | tcc__started                 |      277
 tcc__objoff_pkey                  | tcc__objoff                  |      267
 tcc__rotpos_pkey                  | tcc__rotpos                  |      267
 tcc__rottype_pkey                 | tcc__rottype                 |      263
 tcc__modu_pkey                    | tcc__modu                    |      231
 tcc__objname_pkey                 | tcc__objname                 |      188
 tcc__objnetpos_pkey               | tcc__objnetpos               |      187
 tcc__objpos_pkey                  | tcc__objpos                  |      187
 tcc__objsys_pkey                  | tcc__objsys                  |      187
 tcc__objmag_pkey                  | tcc__objmag                  |      187
 tcc__moveitems_pkey               | tcc__moveitems               |      187
 tcc__objzpmpos_pkey               | tcc__objzpmpos               |      185
 tcc__objinstang_pkey              | tcc__objinstang              |      182
 tcc__caliboff_pkey                | tcc__caliboff                |      179
 tcc__objarcoff_pkey               | tcc__objarcoff               |      177
 tcc__boresight_pkey               | tcc__boresight               |      177
 tcc__guideoff_pkey                | tcc__guideoff                |      177
 tcc__cmddtime_pkey                | tcc__cmddtime                |      172
 tcc__objpm_pkey                   | tcc__objpm                   |      170
 tcc__moved_pkey                   | tcc__moved                   |      170
 tcc__gcamcmd_pkey                 | tcc__gcamcmd                 |       98
 tcc__ut1_pkey                     | tcc__ut1                     |       98
 tcc__utc_tai_pkey                 | tcc__utc_tai                 |       97
 tcc__secactmount_pkey             | tcc__secactmount             |       95
 tcc__secorient_pkey               | tcc__secorient               |       95
 tcc__secdesorient_pkey            | tcc__secdesorient            |       94
 tcc__primcmdmount_pkey            | tcc__primcmdmount            |       94
 tcc__primactmount_pkey            | tcc__primactmount            |       94
 tcc__primorient_pkey              | tcc__primorient              |       94
 tcc__primconstrms_pkey            | tcc__primconstrms            |       94
 tcc__secconstrms_pkey             | tcc__secconstrms             |       94
 tcc__lst_pkey                     | tcc__lst                     |       93
 tcc__primdesorient_pkey           | tcc__primdesorient           |       93
 tcc__tlapse_pkey                  | tcc__tlapse                  |       92
 tcc__timestamp_pkey               | tcc__timestamp               |       92
 tcc__winddir_pkey                 | tcc__winddir                 |       92
 tcc__secfocus_pkey                | tcc__secfocus                |       92
 tcc__windspeed_pkey               | tcc__windspeed               |       92
 tcc__airtemp_pkey                 | tcc__airtemp                 |       91
 tcc__tai_pkey                     | tcc__tai                     |       91
 tcc__sectrusstemp_pkey            | tcc__sectrusstemp            |       91
 tcc__pressure_pkey                | tcc__pressure                |       88
 tcc__humidity_pkey                | tcc__humidity                |       88
 tcc__primf_bftemp_pkey            | tcc__primf_bftemp            |       88
 tcc__secf_bftemp_pkey             | tcc__secf_bftemp             |       88
 tcc__seccmdmount_pkey             | tcc__seccmdmount             |       87
 tcc__telmount_pkey                | tcc__telmount                |       86
 tcc__rotmount_pkey                | tcc__rotmount                |       86
 tcc__gcfocus_pkey                 | tcc__gcfocus                 |       86
 tcc__axiscmdstate_pkey            | tcc__axiscmdstate            |       17
 tcc__currarcoff_pkey              | tcc__currarcoff              |       10
 tcc__axiserrcode_pkey             | tcc__axiserrcode             |        9
 tcc__usernum_pkey                 | tcc__usernum                 |        8
 tcc__scalefac_pkey                | tcc__scalefac                |        6
 tcc__etext_pkey                   | tcc__etext                   |        6
 tcc__slewbeg_pkey                 | tcc__slewbeg                 |        6
 tcc__slewadvtime_pkey             | tcc__slewadvtime             |        6
 tcc__slewduration_pkey            | tcc__slewduration            |        6
 tcc__slewend_pkey                 | tcc__slewend                 |        6
 mcp__controllamps_pkey            | mcp__controllamps            |        6
 tcc__received_pkey                | tcc__received                |        5
 mcp__userid_pkey                  | mcp__userid                  |        5
 mcp__ffscommandedon_pkey          | mcp__ffscommandedon          |        4
 mcp__ffsstatus_pkey               | mcp__ffsstatus               |        4
 mcp__rotbadfiducialdelta_pkey     | mcp__rotbadfiducialdelta     |        4
 mcp__fflamp_pkey                  | mcp__fflamp                  |        4
 tcc__axisinit_pkey                | tcc__axisinit                |        4
 tcc__msgslost_pkey                | tcc__msgslost                |        4
 mcp__fflampcommandedon_pkey       | mcp__fflampcommandedon       |        4
 mcp__semaphoreowner_pkey          | mcp__semaphoreowner          |        4
 tcc__failed_pkey                  | tcc__failed                  |        2
 tcc__warnrotstatus_pkey           | tcc__warnrotstatus           |        2
 tcc__facil_pkey                   | tcc__facil                   |        2
 tcc__slewsuperseded_pkey          | tcc__slewsuperseded          |        2
 tcc__rejectedaxiserrcode_pkey     | tcc__rejectedaxiserrcode     |        2
 tcc__expected_pkey                | tcc__expected                |        2
 tcc__inst_pkey                    | tcc__inst                    |        2
 tcc__instpos_pkey                 | tcc__instpos                 |        2
 tcc__vmsmsg_pkey                  | tcc__vmsmsg                  |        2
 tcc__useradded_pkey               | tcc__useradded               |        2
 tcc__warnaltstatus_pkey           | tcc__warnaltstatus           |        2
 tcc__warnazstatus_pkey            | tcc__warnazstatus            |        2
 tcc__userinfo_pkey                | tcc__userinfo                |        2
 tcc__baduser_pkey                 | tcc__baduser                 |        2
 tcc__azaltdist_pkey               | tcc__azaltdist               |        2
 tcc__ptrefpos_pkey                | tcc__ptrefpos                |        2
 tcc__rotwrappref_pkey             | tcc__rotwrappref             |        2
 tcc__axisnoslew_pkey              | tcc__axisnoslew              |        2
 mcp__needsemaphore_pkey           | mcp__needsemaphore           |        2
 tcc__userdeleted_pkey             | tcc__userdeleted             |        2
 tcc__axisstop_pkey                | tcc__axisstop                |        2
 tcc__secstatus_pkey               | tcc__secstatus               |        2
 tcc__iimscale_pkey                | tcc__iimscale                |        2
 tcc__iimctr_pkey                  | tcc__iimctr                  |        2
 tcc__iimlim_pkey                  | tcc__iimlim                  |        2
 tcc__gimscale_pkey                | tcc__gimscale                |        2
 tcc__gimctr_pkey                  | tcc__gimctr                  |        2
 tcc__gimlim_pkey                  | tcc__gimlim                  |        2
 tcc__rotlim_pkey                  | tcc__rotlim                  |        2
 tcc__gcnfilt_pkey                 | tcc__gcnfilt                 |        2
 mcp__nelamp_pkey                  | mcp__nelamp                  |        2
 mcp__nelampcommandedon_pkey       | mcp__nelampcommandedon       |        2
 mcp__hgcdlampcommandedon_pkey     | mcp__hgcdlampcommandedon     |        2
 tcc__primdesorientage_pkey        | tcc__primdesorientage        |        2
 tcc__primstatus_pkey              | tcc__primstatus              |        2
 tcc__ipconfig_pkey                | tcc__ipconfig                |        2
 mcp__hgcdlamp_pkey                | mcp__hgcdlamp                |        2
 tcc__secdesorientage_pkey         | tcc__secdesorientage         |        2
 tcc__gcview_pkey                  | tcc__gcview                  |        2
 tcc__badazdtime_pkey              | tcc__badazdtime              |        1
 pg_toast_16386_index              | pg_toast_16386               |        1
 pg_toast_16404_index              | pg_toast_16404               |        1
 pg_toast_16410_index              | pg_toast_16410               |        1
 hub__commanders_pkey              | hub__commanders              |        1
 actors_pkey                       | actors                       |        1
 pg_toast_16420_index              | pg_toast_16420               |        1
 pg_toast_16426_index              | pg_toast_16426               |        1
 pg_toast_16453_index              | pg_toast_16453               |        1
 pg_toast_16516_index              | pg_toast_16516               |        1
 pg_toast_16534_index              | pg_toast_16534               |        1
 pg_toast_16449_index              | pg_toast_16449               |        1
 pg_toast_16472_index              | pg_toast_16472               |        1
 pg_toast_16508_index              | pg_toast_16508               |        1
 pg_toast_16573_index              | pg_toast_16573               |        1
 pg_toast_16591_index              | pg_toast_16591               |        1
 pg_toast_16475_index              | pg_toast_16475               |        1
 pg_toast_16562_index              | pg_toast_16562               |        1
 pg_toast_16588_index              | pg_toast_16588               |        1
 pg_toast_16609_index              | pg_toast_16609               |        1
 pg_toast_16697_index              | pg_toast_16697               |        1
 pg_toast_16700_index              | pg_toast_16700               |        1
 pg_toast_16723_index              | pg_toast_16723               |        1
 pg_toast_16761_index              | pg_toast_16761               |        1
 pg_toast_16869_index              | pg_toast_16869               |        1
 pg_toast_16917_index              | pg_toast_16917               |        1
 pg_toast_17015_index              | pg_toast_17015               |        1
 pg_toast_17033_index              | pg_toast_17033               |        1
 pg_toast_17041_index              | pg_toast_17041               |        1
 pg_toast_17059_index              | pg_toast_17059               |        1
 pg_toast_17082_index              | pg_toast_17082               |        1
 tcc__axisnotrack_pkey             | tcc__axisnotrack             |        1
 pg_toast_17090_index              | pg_toast_17090               |        1
 tcc__schmovetimes_pkey            | tcc__schmovetimes            |        1
 mcp__ffsmoved_pkey                | mcp__ffsmoved                |        1
 mcp__ffsopenfailed_pkey           | mcp__ffsopenfailed           |        1
 pg_toast_17163_index              | pg_toast_17163               |        1
 pg_toast_17223_index              | pg_toast_17223               |        1
 perms__authlist_pkey              | perms__authlist              |        1
 mcp__saddleismounted_pkey         | mcp__saddleismounted         |        1
 pg_toast_17274_index              | pg_toast_17274               |        1
 mcp__altfiducialversion_pkey      | mcp__altfiducialversion      |        1
 mcp__minencodermismatch_pkey      | mcp__minencodermismatch      |        1
 pg_toast_17176_index              | pg_toast_17176               |        1
 pg_toast_17221_index              | pg_toast_17221               |        1
 perms__actors_pkey                | perms__actors                |        1
 mcp__ffsselected_pkey             | mcp__ffsselected             |        1
 pg_toast_17280_index              | pg_toast_17280               |        1
 mcp__azfiducialversion_pkey       | mcp__azfiducialversion       |        1
 mcp__plcversion_pkey              | mcp__plcversion              |        1
 mcp__cwpositions_pkey             | mcp__cwpositions             |        1
 mcp__rotmaxaccrequested_pkey      | mcp__rotmaxaccrequested      |        1
 pg_toast_17220_index              | pg_toast_17220               |        1
 perms__programs_pkey              | perms__programs              |        1
 mcp__whtlampcommandedon_pkey      | mcp__whtlampcommandedon      |        1
 mcp__instrumentnum_pkey           | mcp__instrumentnum           |        1
 mcp__msonmaxcorrection_pkey       | mcp__msonmaxcorrection       |        1
 pg_toast_17204_index              | pg_toast_17204               |        1
 hub__users_pkey                   | hub__users                   |        1
 mcp__instrumentnumconsistent_pkey | mcp__instrumentnumconsistent |        1
 mcp__goodfiducialversions_pkey    | mcp__goodfiducialversions    |        1
 mcp__cwstatus_pkey                | mcp__cwstatus                |        1
 pg_toast_17212_index              | pg_toast_17212               |        1
 hub__user_pkey                    | hub__user                    |        1
 mcp__uvlampcommandedon_pkey       | mcp__uvlampcommandedon       |        1
 pg_toast_17283_index              | pg_toast_17283               |        1
 mcp__rotfiducialversion_pkey      | mcp__rotfiducialversion      |        1
 pg_toast_17318_index              | pg_toast_17318               |        1
 mcp__mcpversion_pkey              | mcp__mcpversion              |        1
 mcp__lampscommanded_pkey          | mcp__lampscommanded          |        1
 mcp__azmson_pkey                  | mcp__azmson                  |        1
 mcp__altmson_pkey                 | mcp__altmson                 |        1
 mcp__rotmson_pkey                 | mcp__rotmson                 |        1
 mcp__ffscommanded_pkey            | mcp__ffscommanded            |        1
 mcp__azmaxaccrequested_pkey       | mcp__azmaxaccrequested       |        1
 pg_toast_17366_index              | pg_toast_17366               |        1
 mcp__text_pkey                    | mcp__text                    |        1
 tcc__exptime_pkey                 | tcc__exptime                 |        1
 tcc__superseded_pkey              | tcc__superseded              |        1
 pg_toast_17801_index              | pg_toast_17801               |        1
 hub__text_pkey                    | hub__text                    |        1
 pg_toast_17809_index              | pg_toast_17809               |        1
 hub__actors_pkey                  | hub__actors                  |        1
 pg_toast_16389_index              | pg_toast_16389               |        1
(223 rows)

Next, look at the archive server log file. First, look for unknown keywords:

[dkirkby@hub25m archiver-28070]$ fgrep "Unknown keyword" server.log | sort -uk 4
2009-05-05 08:23:36-0600 [ReplyReceiver,client] 'Unknown keyword hub.hubtxt'
2009-05-26 10:20:10-0600 [ReplyReceiver,client] 'Unknown keyword hub.parseerror'
2009-05-26 08:07:56-0600 [ReplyReceiver,client] 'Unknown keyword hub.version'
2009-04-28 19:00:01-0600 [ReplyReceiver,client] 'Unknown keyword mcp.aliveat'
2009-04-28 20:19:06-0600 [ReplyReceiver,client] 'Unknown keyword mcp.altmsoff'
2009-04-28 20:19:04-0600 [ReplyReceiver,client] 'Unknown keyword mcp.azmsoff'
2009-04-29 17:35:35-0600 [ReplyReceiver,client] 'Unknown keyword mcp.keywordqueuedepth'
2009-05-05 08:23:14-0600 [ReplyReceiver,client] 'Unknown keyword mcp.lavalamp'
2009-04-28 20:19:07-0600 [ReplyReceiver,client] 'Unknown keyword mcp.rotmsoff'
2009-04-28 20:19:04-0600 [ReplyReceiver,client] 'Unknown keyword mcp.trace'
2009-06-06 04:16:45-0600 [ReplyReceiver,client] 'Unknown keyword tcc.axishighaccel'
2009-05-05 20:00:05-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gcamid'
2009-05-05 20:00:05-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gcaminfo'
2009-05-05 20:00:24-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gcamreply'
2009-05-14 22:12:11-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gpctr'
2009-05-15 01:26:41-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gpfindstar'
2009-05-14 22:12:11-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gplim'
2009-05-14 22:12:11-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gprobe'
2009-05-14 22:12:11-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gprotgimang'
2009-05-14 22:12:11-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gprotxy'
2009-05-16 02:01:52-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gresid'
2009-05-15 00:53:20-0600 [ReplyReceiver,client] 'Unknown keyword tcc.gsdatapix'
2009-05-15 01:26:58-0600 [ReplyReceiver,client] 'Unknown keyword tcc.nostarsfound'
2009-05-10 18:47:45-0600 [ReplyReceiver,client] 'Unknown keyword tcc.slewends'

Here are the newly created tables:

[dkirkby@hub25m archiver-28070]$ fgrep "creating" server.log
2009-04-28 20:23:36-0600 [ReplyReceiver,client] creating new table mcp__ffsstatus with ['raw_id__int8', 'val1_0__int2', 'val1_1__int2', 'val1_2__int2', 'val1_3__int2', 'val1_4__int2', 'val1_5__int2', 'val1_6__int2', 'val1_7__int2']
2009-04-28 20:23:36-0600 [ReplyReceiver,client] creating new table mcp__ffscommandedon with ['raw_id__int8', 'val1__int2']
2009-04-28 20:38:51-0600 [ReplyReceiver,client] creating new table mcp__needsemaphore with ['raw_id__int8']
2009-04-28 20:38:53-0600 [ReplyReceiver,client] creating new table tcc__axisnotrack with ['raw_id__int8', 'val1__text']
2009-04-28 20:38:57-0600 [ReplyReceiver,client] creating new table tcc__axisnoslew with ['raw_id__int8', 'val1__text']
2009-04-28 20:39:20-0600 [ReplyReceiver,client] creating new table tcc__schmovetimes with ['raw_id__int8', 'val1_0__flt4', 'val1_1__flt4', 'val1_2__flt4', 'val1_3__flt4', 'val1_4__flt4', 'val1_5__flt4', 'val1_6__flt4', 'val1_7__flt4', 'val1_8__flt4', 'val1_9__flt4']
2009-04-28 20:40:48-0600 [ReplyReceiver,client] creating new table tcc__objpm with ['raw_id__int8', 'val1_0__flt4', 'val1_1__flt4', 'val2__flt4', 'val3__flt4']
2009-04-28 20:40:48-0600 [ReplyReceiver,client] creating new table tcc__rotwrappref with ['raw_id__int8', 'val1__int2']
2009-04-28 20:44:00-0600 [ReplyReceiver,client] creating new table tcc__ptrefpos with ['raw_id__int8', 'val1__flt4', 'val2__flt4', 'val3__flt8', 'val4__flt4', 'val5__flt4', 'val6__flt8']
2009-04-28 20:44:00-0600 [ReplyReceiver,client] creating new table tcc__azaltdist with ['raw_id__int8', 'val1__flt4']
2009-04-28 20:44:52-0600 [ReplyReceiver,client] creating new table tcc__moved with ['raw_id__int8']
2009-04-28 20:46:13-0600 [ReplyReceiver,client] creating new table tcc__telmount with ['raw_id__int8', 'val1__flt4', 'val2__flt4', 'val3__flt8', 'val4__flt4', 'val5__flt4', 'val6__flt8']
2009-04-28 20:46:13-0600 [ReplyReceiver,client] creating new table tcc__rotmount with ['raw_id__int8', 'val1__flt4', 'val2__flt4', 'val3__flt8']
2009-04-29 05:34:51-0600 [ReplyReceiver,client] creating new table tcc__msgslost with ['raw_id__int8', 'val1__int4']
2009-04-29 06:00:44-0600 [ReplyReceiver,client] creating new table tcc__baduser with ['raw_id__int8', 'val1__int4']
2009-04-29 06:00:44-0600 [ReplyReceiver,client] creating new table tcc__userinfo with ['raw_id__int8', 'val1__int4', 'val2__int4', 'val3__int4', 'val4__int4', 'val5_0__int4', 'val5_1__int4']
2009-04-29 11:31:09-0600 [ReplyReceiver,client] creating new table mcp__ffsmoved with ['raw_id__int8', 'val1__int2']
2009-04-29 11:31:09-0600 [ReplyReceiver,client] creating new table mcp__ffsopenfailed with ['raw_id__int8', 'val1__int2']
2009-04-29 17:38:37-0600 [ReplyReceiver,client] creating new table tcc__expected with ['raw_id__int8', 'val1__text']
2009-04-29 20:01:22-0600 [ReplyReceiver,client] creating new table tcc__slewsuperseded with ['raw_id__int8']
2009-05-01 17:18:57-0600 [ReplyReceiver,client] creating new table tcc__rejectedaxiserrcode with ['raw_id__int8', 'val1_0__text', 'val1_1__text', 'val1_2__text']
2009-05-02 21:20:00-0600 [ReplyReceiver,client] creating new table tcc__warnaltstatus with ['raw_id__int8']
2009-05-02 23:10:06-0600 [ReplyReceiver,client] creating new table tcc__warnrotstatus with ['raw_id__int8']
2009-05-02 23:22:24-0600 [ReplyReceiver,client] creating new table mcp__rotmaxaccrequested with ['raw_id__int8', 'val1__flt4']
2009-05-03 00:35:20-0600 [ReplyReceiver,client] creating new table tcc__warnazstatus with ['raw_id__int8']
2009-05-05 08:22:58-0600 [ReplyReceiver,client] creating new table hub__users with ['raw_id__int8', 'val1_0__text']
2009-05-05 08:22:58-0600 [ReplyReceiver,client] creating new table hub__user with ['raw_id__int8', 'val1_0__text', 'val1_1__text', 'val1_2__text', 'val1_3__text', 'val1_4__text', 'val1_5__text']
2009-05-05 08:22:59-0600 [ReplyReceiver,client] creating new table perms__actors with ['raw_id__int8', 'val1_0__text']
2009-05-05 08:22:59-0600 [ReplyReceiver,client] creating new table perms__programs with ['raw_id__int8', 'val1_0__text']
2009-05-05 08:22:59-0600 [ReplyReceiver,client] creating new table perms__authlist with ['raw_id__int8', 'val1_0__text']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__ffsselected with ['raw_id__int8', 'val1__int2']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__uvlampcommandedon with ['raw_id__int8', 'val1__int2']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__whtlampcommandedon with ['raw_id__int8', 'val1__int2']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__saddleismounted with ['raw_id__int8', 'val1__int2']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__instrumentnumconsistent with ['raw_id__int8', 'val1__int2']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__instrumentnum with ['raw_id__int8', 'val1__int4']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__azfiducialversion with ['raw_id__int8', 'val1__text']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__altfiducialversion with ['raw_id__int8', 'val1__text']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__rotfiducialversion with ['raw_id__int8', 'val1__text']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__goodfiducialversions with ['raw_id__int8', 'val1__int2']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__msonmaxcorrection with ['raw_id__int8', 'val1_0__int4', 'val1_1__int4', 'val1_2__int4']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__minencodermismatch with ['raw_id__int8', 'val1_0__int4', 'val1_1__int4', 'val1_2__int4']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__plcversion with ['raw_id__int8', 'val1__int4']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__mcpversion with ['raw_id__int8', 'val1__text']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__cwpositions with ['raw_id__int8', 'val1_0__int4', 'val1_1__int4', 'val1_2__int4', 'val1_3__int4']
2009-05-05 08:23:14-0600 [ReplyReceiver,client] creating new table mcp__cwstatus with ['raw_id__int8', 'val1_0__int2', 'val1_1__int2', 'val1_2__int2', 'val1_3__int2']
2009-05-05 17:40:15-0600 [ReplyReceiver,client] creating new table mcp__lampscommanded with ['raw_id__int8', 'val1__int2']
2009-05-05 18:30:25-0600 [ReplyReceiver,client] creating new table mcp__azmson with ['raw_id__int8', 'val1__int2']
2009-05-05 18:30:25-0600 [ReplyReceiver,client] creating new table mcp__altmson with ['raw_id__int8', 'val1__int2']
2009-05-05 18:30:26-0600 [ReplyReceiver,client] creating new table mcp__rotmson with ['raw_id__int8', 'val1__int2']
2009-05-05 19:55:47-0600 [ReplyReceiver,client] creating new table mcp__ffscommanded with ['raw_id__int8', 'val1__int2']
2009-05-05 21:18:17-0600 [ReplyReceiver,client] creating new table mcp__azmaxaccrequested with ['raw_id__int8', 'val1__flt4']
2009-05-06 12:35:05-0600 [ReplyReceiver,client] creating new table mcp__text with ['raw_id__int8', 'val1__text']
2009-05-15 01:16:46-0600 [ReplyReceiver,client] creating new table tcc__exptime with ['raw_id__int8', 'val1__flt4']
2009-05-15 02:23:45-0600 [ReplyReceiver,client] creating new table tcc__superseded with ['raw_id__int8']
2009-05-26 08:00:55-0600 [ReplyReceiver,client] creating new table hub__text with ['raw_id__int8', 'val1__text']
2009-05-26 08:07:56-0600 [ReplyReceiver,client] creating new table hub__actors with ['raw_id__int8', 'val1_0__text']
2009-06-02 11:01:34-0600 [ReplyReceiver,client] creating new table tcc__badazdtime with ['raw_id__int8']

There seems to be something wrong with the reporting of idle times for flushed tables, for example:

Is this a problem with the actual idle times being used or just a problem with how it is displayed in the log file?

Copy the server log to my home directory in case hub25m is rebooted.

Look at HTTPD activity:

[dkirkby@hub25m archiver-28070]$ fgrep HTTP server.log | awk '{print $1,$2,$4,$9,$10,$11,$12,$13,$14}'
2009-04-29 01:36:44-0600 121.30.225.236 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-04-29 16:20:37-0600 128.200.48.167 "GET /archiver/info/ HTTP/1.1" 200 5857 "-"
2009-04-29 16:20:37-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info/"
2009-04-29 16:20:37-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "-"
2009-04-29 16:20:40-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "-"
2009-04-29 16:21:32-0600 128.200.48.167 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info/"
2009-04-29 16:21:35-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5857 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-04-29 16:56:07-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5857 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-04-29 16:56:07-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-29 17:25:01-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5845 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-04-29 17:25:02-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-29 18:54:18-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5845 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-04-29 18:54:18-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 08:34:58-0600 68.4.3.70 "GET /archiver/info HTTP/1.1" 200 5871 "-"
2009-04-30 08:34:58-0600 68.4.3.70 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 08:34:58-0600 68.4.3.70 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 12:34:37-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5871 "-"
2009-04-30 12:34:37-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 12:34:37-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 17:40:45-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5872 "-"
2009-04-30 17:40:45-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 17:40:45-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 19:16:12-0600 68.4.3.70 "GET /archiver/info HTTP/1.1" 200 5859 "-"
2009-04-30 19:16:12-0600 68.4.3.70 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-04-30 19:16:12-0600 68.4.3.70 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-04 12:14:10-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5869 "-"
2009-05-04 12:14:10-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-04 12:14:10-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "-"
2009-05-04 12:14:13-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "-"
2009-05-04 12:14:27-0600 128.200.48.167 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-04 12:14:31-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 5869 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 04:50:31-0600 173.72.49.61 "GET / HTTP/1.1" 404 173 "-"
2009-05-05 04:50:32-0600 173.72.49.61 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m:8080/"
2009-05-05 04:50:58-0600 173.72.49.61 "GET / HTTP/1.1" 404 173 "-"
2009-05-05 13:32:48-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 13:32:48-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 13:32:48-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 304 - "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 13:38:42-0600 128.200.48.167 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 13:38:44-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:11:51-0600 140.180.34.76 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 14:11:52-0600 140.180.34.76 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:11:52-0600 140.180.34.76 "GET /favicon.ico HTTP/1.1" 404 173 "-"
2009-05-05 14:11:55-0600 140.180.34.76 "GET /favicon.ico HTTP/1.1" 404 173 "-"
2009-05-05 14:11:58-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 14:11:59-0600 128.95.99.44 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:11:59-0600 128.95.99.44 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:12:14-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:12:19-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:13:04-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 14:13:07-0600 140.180.34.76 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 14:13:14-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:13:18-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:16:32-0600 128.200.48.167 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:16:47-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 1048 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:16:53-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:16:56-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:17:40-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:17:43-0600 128.95.99.44 "GET /archiver HTTP/1.1" 405 250 "-"
2009-05-05 14:17:47-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 14:17:48-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:17:52-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:17:54-0600 128.200.48.167 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:17:54-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:17:57-0600 128.95.99.44 "GET /archiver/ HTTP/1.1" 404 173 "-"
2009-05-05 14:17:58-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:18:01-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "-"
2009-05-05 14:18:07-0600 128.200.48.167 "GET /archiver/ HTTP/1.1" 404 173 "-"
2009-05-05 14:18:39-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:18:42-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-05 14:18:46-0600 128.95.99.44 "GET /archiver/log HTTP/1.1" 200 921 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-05 14:18:48-0600 128.95.99.44 "GET /archiver/info HTTP/1.1" 200 6235 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-05-08 08:53:26-0600 91.121.116.46 "GET /jmx-console/?x=x HTTP/1.0" 404 173 "-"
2009-05-09 03:48:56-0600 221.195.40.169 "GET http://www.yahoo.com/ HTTP/1.1" 404 173 "-"
2009-05-10 02:40:47-0600 77.37.205.147 "GET http://www.ip2location.com/ HTTP/1.1" 404 173 "-"
2009-05-12 13:21:35-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6240 "-"
2009-05-12 13:21:35-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-12 13:21:35-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 304 - "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-12 13:35:27-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6240 "-"
2009-05-12 13:35:27-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-12 13:35:27-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-05-12 15:42:34-0600 77.37.179.175 "GET http://www.ip2location.com/ HTTP/1.1" 404 173 "-"
2009-05-13 00:07:24-0600 125.230.146.150 "CONNECT 203.188.201.253:25 HTTP/1.1" 404 173 "-"
2009-05-22 08:41:13-0600 60.190.59.240 "GET http://www.yahoo.com/ HTTP/1.1" 404 173 "-"
2009-06-01 10:51:05-0600 125.230.146.206 "CONNECT 203.188.201.253:25 HTTP/1.1" 404 173 "-"
2009-06-01 14:53:25-0600 165.138.116.27 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-06-02 13:43:26-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6283 "-"
2009-06-02 13:43:26-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-06-02 13:43:26-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-06-02 13:57:22-0600 128.200.48.167 "GET /archiver/log HTTP/1.1" 200 11228 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-06-02 13:57:28-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6283 "http://hub25m.apo.nmsu.edu:8080/archiver/log"
2009-06-03 03:10:26-0600 201.116.19.164 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-06-04 13:18:23-0600 91.93.106.30 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-06-06 14:15:04-0600 193.136.119.11 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-06-07 11:35:39-0600 155.44.218.141 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-06-07 15:23:16-0600 193.95.192.9 "GET http://www.yahoo.com/ HTTP/1.1" 404 173 "-"
2009-06-07 15:25:26-0600 220.162.207.62 "GET http://www.yahoo.com/ HTTP/1.1" 404 173 "-"
2009-06-07 22:44:50-0600 201.24.208.40 "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html"
2009-06-09 12:05:22-0600 128.200.48.167 "GET /archiver/info HTTP/1.1" 200 6285 "-"
2009-06-09 12:05:23-0600 128.200.48.167 "GET /static/styles.css HTTP/1.1" 200 820 "http://hub25m.apo.nmsu.edu:8080/archiver/info"
2009-06-09 12:05:23-0600 128.200.48.167 "GET /favicon.ico HTTP/1.1" 404 173 "http://hub25m.apo.nmsu.edu:8080/archiver/info"

Who is requesting /manager/html? The request comes from different hosts but always a Mozilla/4.0 browser:

[dkirkby@hub25m archiver-28070]$ fgrep /manager/html server.log
2009-04-29 01:36:44-0600 [HTTPChannel,0,121.30.225.236] 121.30.225.236 - - [29/Apr/2009:07:36:44 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"
2009-06-01 14:53:25-0600 [HTTPChannel,53,165.138.116.27] 165.138.116.27 - - [01/Jun/2009:20:53:24 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"
2009-06-03 03:10:26-0600 [HTTPChannel,57,201.116.19.164] 201.116.19.164 - - [03/Jun/2009:09:10:25 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"
2009-06-04 13:18:23-0600 [HTTPChannel,58,91.93.106.30] 91.93.106.30 - - [04/Jun/2009:19:18:22 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"
2009-06-06 14:15:04-0600 [HTTPChannel,59,193.136.119.11] 193.136.119.11 - - [06/Jun/2009:20:15:03 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"
2009-06-07 11:35:39-0600 [HTTPChannel,60,155.44.218.141] 155.44.218.141 - - [07/Jun/2009:17:35:38 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"
2009-06-07 22:44:50-0600 [HTTPChannel,63,201.24.208.40] 201.24.208.40 - - [08/Jun/2009:04:44:49 +0000] "GET /manager/html HTTP/1.1" 404 173 "http://192.41.211.191:8080/manager/html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE 3.01)"

This seems to be related to Apache Tomcat. The different hosts look suspicious… is this an exploit attempt?

Why is the /archiver/info page triggering a /favicon.ico request? I don’t think I configured that, but perhaps some browsers do it automatically?

Some other odd requests:

2009-05-08 08:53:26-0600 91.121.116.46 "GET /jmx-console/?x=x HTTP/1.0" 404 173 "-"
2009-05-09 03:48:56-0600 221.195.40.169 "GET http://www.yahoo.com/ HTTP/1.1" 404 173 "-"
2009-05-10 02:40:47-0600 77.37.205.147 "GET http://www.ip2location.com/ HTTP/1.1" 404 173 "-"
2009-05-13 00:07:24-0600 125.230.146.150 "CONNECT 203.188.201.253:25 HTTP/1.1" 404 173 "-"

Jon B has setup the privs so I can see the postgres db files. The layout is documented here.

[dkirkby@hub25m base]$ ls /var/lib/pgsql/data/base
1  10792  10793

Why are there only subdirs for 3 databases?? Which databases are these? The pg_stat_databases view lists 6. Even stranger, the contents of each subdir appear identical:

[dkirkby@hub25m base]$ ls 1
10287  10295  10302  10307  10723  10728  10733  10738  10743  10748  1247  1250  2600  2603  2606  2609  2612  2615  2618  2650  2653  2656  2659  2662  2665  2668  2673  2678  2681  2684  2687  2690  2693  2700  2703
10289  10299  10304  10308  10725  10730  10735  10740  10745  10750  1248  1255  2601  2604  2607  2610  2613  2616  2619  2651  2654  2657  2660  2663  2666  2669  2674  2679  2682  2685  2688  2691  2696  2701  2704
10293  10301  10305  10310  10727  10732  10737  10742  10747  10752  1249  1259  2602  2605  2608  2611  2614  2617  2620  2652  2655  2658  2661  2664  2667  2670  2675  2680  2683  2686  2689  2692  2699  2702  PG_VERSION
[dkirkby@hub25m base]$ ls 10792
10287  10295  10302  10307  10723  10728  10733  10738  10743  10748  1247  1250  2600  2603  2606  2609  2612  2615  2618  2650  2653  2656  2659  2662  2665  2668  2673  2678  2681  2684  2687  2690  2693  2700  2703
10289  10299  10304  10308  10725  10730  10735  10740  10745  10750  1248  1255  2601  2604  2607  2610  2613  2616  2619  2651  2654  2657  2660  2663  2666  2669  2674  2679  2682  2685  2688  2691  2696  2701  2704
10293  10301  10305  10310  10727  10732  10737  10742  10747  10752  1249  1259  2602  2605  2608  2611  2614  2617  2620  2652  2655  2658  2661  2664  2667  2670  2675  2680  2683  2686  2689  2692  2699  2702  PG_VERSION
[dkirkby@hub25m base]$ ls 10793
10287  10295  10302  10307  10723  10728  10733  10738  10743  10748  1247  1250  2600  2603  2606  2609  2612  2615  2618  2650  2653  2656  2659  2662  2665  2668  2673  2678  2681  2684  2687  2690  2693  2700  2703
10289  10299  10304  10308  10725  10730  10735  10740  10745  10750  1248  1255  2601  2604  2607  2610  2613  2616  2619  2651  2654  2657  2660  2663  2666  2669  2674  2679  2682  2685  2688  2691  2696  2701  2704
10293  10301  10305  10310  10727  10732  10737  10742  10747  10752  1249  1259  2602  2605  2608  2611  2614  2617  2620  2652  2655  2658  2661  2664  2667  2670  2675  2680  2683  2686  2689  2692  2699  2702  PG_VERSION

The numbers within this subdirs should correspond to the relfilenode values I obtained above. For example, reply_raw has relfilenode=16386 but this doesn’t appear here!? At least I can confirm that the page size is 8192 bytes:

[dkirkby@hub25m base]$ ls -l 1
total 4148
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10287
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10289
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10293
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10295
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10299
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10301
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10302
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10304
-rw-r--r-- 1 26 26 114688 Apr  2 19:23 10305
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 10307
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10308
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10310
-rw-r--r-- 1 26 26  49152 Apr  2 19:23 10723
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10725
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10727
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10728
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10730
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10732
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10733
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10735
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10737
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10738
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10740
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10742
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10743
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10745
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10747
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10748
-rw-r--r-- 1 26 26      0 Apr  2 19:23 10750
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 10752
-rw-r--r-- 1 26 26  40960 Apr  2 19:23 1247
-rw-r--r-- 1 26 26      0 Apr  2 19:23 1248
-rw-r--r-- 1 26 26 204800 Apr  2 19:23 1249
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 1250
-rw-r--r-- 1 26 26 360448 Apr  2 19:23 1255
-rw-r--r-- 1 26 26  40960 Apr  2 19:23 1259
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2600
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2601
-rw-r--r-- 1 26 26  24576 Apr  2 19:23 2602
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2603
-rw-r--r-- 1 26 26      0 Apr  2 19:23 2604
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2605
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2606
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2607
-rw-r--r-- 1 26 26 270336 Apr  2 19:23 2608
-rw-r--r-- 1 26 26 131072 Apr  2 19:23 2609
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2610
-rw-r--r-- 1 26 26      0 Apr  2 19:23 2611
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2612
-rw-r--r-- 1 26 26      0 Apr  2 19:23 2613
-rw-r--r-- 1 26 26      0 Apr  2 19:23 2614
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2615
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2616
-rw-r--r-- 1 26 26 114688 Apr  2 19:23 2617
-rw-r--r-- 1 26 26  65536 Apr  2 19:23 2618
-rw-r--r-- 1 26 26  65536 Apr  2 19:23 2619
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2620
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2650
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2651
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2652
-rw-r--r-- 1 26 26  32768 Apr  2 19:23 2653
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2654
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2655
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2656
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2657
-rw-r--r-- 1 26 26 212992 Apr  2 19:23 2658
-rw-r--r-- 1 26 26  57344 Apr  2 19:23 2659
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2660
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2661
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2662
-rw-r--r-- 1 26 26  73728 Apr  2 19:23 2663
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2664
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2665
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2666
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2667
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2668
-rw-r--r-- 1 26 26  32768 Apr  2 19:23 2669
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2670
-rw-r--r-- 1 26 26 180224 Apr  2 19:23 2673
-rw-r--r-- 1 26 26 204800 Apr  2 19:23 2674
-rw-r--r-- 1 26 26  81920 Apr  2 19:23 2675
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2678
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2679
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2680
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2681
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2682
-rw-r--r-- 1 26 26   8192 Apr  2 19:23 2683
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2684
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2685
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2686
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2687
-rw-r--r-- 1 26 26  32768 Apr  2 19:23 2688
-rw-r--r-- 1 26 26 106496 Apr  2 19:23 2689
-rw-r--r-- 1 26 26  73728 Apr  2 19:23 2690
-rw-r--r-- 1 26 26 335872 Apr  2 19:23 2691
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2692
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2693
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2696
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2699
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2700
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2701
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2702
-rw-r--r-- 1 26 26  16384 Apr  2 19:23 2703
-rw-r--r-- 1 26 26  40960 Apr  2 19:23 2704
-rw-r--r-- 1 26 26      4 Apr  2 19:23 PG_VERSION

What’s special about the date these were all created (Apr 2)? The postgres server was started 45 days ago and the archiver started 41 days ago. 45 days ago was Apr 25, so I am guess that the server is now configured to save its files somewhere else? I was looking in the wrong place:

test=# show data_directory;
 data_directory     
-----------------------
 /usr/local/pgsql/data
(1 row)

I can’t see into this one either, so write back to Jon…ok here are the real files:

[dkirkby@hub25m data]$ cd /usr/local/pgsql/data/base
[dkirkby@hub25m base]$ du -sk *
4872    1
4872    11510
4968    11511
3331244    16384
4448    17374
12908    17375
4    pgsql_tmp

This shows 3.3Gb in the “test” database (oid 16384), consistent with my 3.1Gb estimate above.

[dkirkby@hub25m base]$ ls -lS 16384
total 3331220
-rw-r--r-- 1 postgres postgres 917782528 Jun 10 15:09 16386
-rw-r--r-- 1 postgres postgres 491569152 Jun 10 15:09 16389
-rw-r--r-- 1 postgres postgres 232013824 Jun 10 15:09 16402
-rw-r--r-- 1 postgres postgres 182870016 Jun 10 15:09 16398
-rw-r--r-- 1 postgres postgres 182870016 Jun 10 15:09 16400
-rw-r--r-- 1 postgres postgres 182870016 Jun 10 15:09 16403
-rw-r--r-- 1 postgres postgres 124035072 May 15 04:25 16697
-rw-r--r-- 1 postgres postgres  54239232 Jun 10 15:09 16463
-rw-r--r-- 1 postgres postgres  54190080 Jun 10 15:09 16472
-rw-r--r-- 1 postgres postgres  54181888 Jun 10 15:09 16444
-rw-r--r-- 1 postgres postgres  54173696 Jun 10 15:09 16441
-rw-r--r-- 1 postgres postgres  54157312 Jun 10 15:09 16449
-rw-r--r-- 1 postgres postgres  54116352 Jun 10 15:09 16453
-rw-r--r-- 1 postgres postgres  39567360 Jun 10 15:09 16436
-rw-r--r-- 1 postgres postgres  39460864 May 15 04:25 16709
-rw-r--r-- 1 postgres postgres  27090944 Jun 10 15:09 16447
-rw-r--r-- 1 postgres postgres  27090944 Jun 10 15:09 16452
-rw-r--r-- 1 postgres postgres  27090944 Jun 10 15:09 16466
-rw-r--r-- 1 postgres postgres  27090944 Jun 10 15:09 16468
-rw-r--r-- 1 postgres postgres  27090944 Jun 10 15:09 16470
-rw-r--r-- 1 postgres postgres  27090944 Jun 10 15:09 16489
-rw-r--r-- 1 postgres postgres  26140672 May 15 04:24 16516
-rw-r--r-- 1 postgres postgres  19709952 Jun 10 15:09 16439
-rw-r--r-- 1 postgres postgres  18407424 Jun  9 01:55 16534
-rw-r--r-- 1 postgres postgres  18194432 Jun  9 02:09 16829
-rw-r--r-- 1 postgres postgres  14729216 Jun  9 02:09 16767
-rw-r--r-- 1 postgres postgres  14704640 Jun  9 02:09 16772
-rw-r--r-- 1 postgres postgres  12648448 Jun  9 02:09 16761
-rw-r--r-- 1 postgres postgres  12451840 May 15 04:24 16522
-rw-r--r-- 1 postgres postgres  11943936 Jun  9 02:14 16420
-rw-r--r-- 1 postgres postgres  10813440 Jun  9 01:56 16683
-rw-r--r-- 1 postgres postgres   8888320 Jun  9 02:09 16832
-rw-r--r-- 1 postgres postgres   8372224 Jun  9 02:09 16780
-rw-r--r-- 1 postgres postgres   8134656 Jun  9 02:09 16792
-rw-r--r-- 1 postgres postgres   8126464 Jun  9 02:09 16807
-rw-r--r-- 1 postgres postgres   7454720 Jun  9 01:49 16637
-rw-r--r-- 1 postgres postgres   6987776 Jun  9 01:55 16540
-rw-r--r-- 1 postgres postgres   6299648 Jun  9 01:55 16657
-rw-r--r-- 1 postgres postgres   6217728 Jun  9 01:49 16652
-rw-r--r-- 1 postgres postgres   6184960 Jun  9 02:09 16789
-rw-r--r-- 1 postgres postgres   6127616 Jun  9 02:09 16798
-rw-r--r-- 1 postgres postgres   6111232 Jun  9 02:09 16770
-rw-r--r-- 1 postgres postgres   6111232 Jun  9 02:09 16775
-rw-r--r-- 1 postgres postgres   6111232 Jun  9 02:09 16787
-rw-r--r-- 1 postgres postgres   6094848 Jun  9 02:09 16817
-rw-r--r-- 1 postgres postgres   6086656 Jun  9 02:09 16820
-rw-r--r-- 1 postgres postgres   5660672 Jun  9 01:55 16671
-rw-r--r-- 1 postgres postgres   5439488 Jun  9 02:14 16426
-rw-r--r-- 1 postgres postgres   5218304 Jun  9 01:54 16668
-rw-r--r-- 1 postgres postgres   5160960 Jun  9 01:49 16630
-rw-r--r-- 1 postgres postgres   5128192 Jun  9 01:49 16625
-rw-r--r-- 1 postgres postgres   5070848 Jun  9 01:49 16646
-rw-r--r-- 1 postgres postgres   4997120 Jun  9 01:49 16640
-rw-r--r-- 1 postgres postgres   4874240 Jun  9 01:56 16660
-rw-r--r-- 1 postgres postgres   4866048 Jun  9 02:09 16814
-rw-r--r-- 1 postgres postgres   4857856 Jun  9 02:09 16777
-rw-r--r-- 1 postgres postgres   4857856 Jun  9 02:09 16795
-rw-r--r-- 1 postgres postgres   4603904 Jun  9 01:35 16917
-rw-r--r-- 1 postgres postgres   4464640 Jun  9 01:55 16690
-rw-r--r-- 1 postgres postgres   4268032 Jun  9 01:39 16557
-rw-r--r-- 1 postgres postgres   4227072 Jun  9 01:34 16723
-rw-r--r-- 1 postgres postgres   3923968 Jun  9 01:54 16682
-rw-r--r-- 1 postgres postgres   3923968 Jun  8 10:34 17103
-rw-r--r-- 1 postgres postgres   3850240 Jun  9 02:14 16432
-rw-r--r-- 1 postgres postgres   3735552 Jun  9 02:09 16834
-rw-r--r-- 1 postgres postgres   3497984 Jun  9 01:49 16615
-rw-r--r-- 1 postgres postgres   3465216 Jun  9 01:49 16609
-rw-r--r-- 1 postgres postgres   3465216 Jun  9 01:49 16620
-rw-r--r-- 1 postgres postgres   2973696 Jun  8 10:34 17123
-rw-r--r-- 1 postgres postgres   2916352 Jun  9 02:09 16785
-rw-r--r-- 1 postgres postgres   2916352 Jun  9 02:09 16812
-rw-r--r-- 1 postgres postgres   2875392 Jun  9 02:09 16803
-rw-r--r-- 1 postgres postgres   2875392 Jun  9 02:09 16805
-rw-r--r-- 1 postgres postgres   2859008 Jun  9 02:09 16810
-rw-r--r-- 1 postgres postgres   2859008 Jun  9 02:09 16825
-rw-r--r-- 1 postgres postgres   2859008 Jun  9 02:09 16826
-rw-r--r-- 1 postgres postgres   2818048 Jun  9 02:09 16746
-rw-r--r-- 1 postgres postgres   2809856 Jun  9 02:09 16783
-rw-r--r-- 1 postgres postgres   2801664 Jun  9 02:09 16864
-rw-r--r-- 1 postgres postgres   2785280 Jun  9 02:09 16849
-rw-r--r-- 1 postgres postgres   2760704 Jun  9 02:09 16801
-rw-r--r-- 1 postgres postgres   2736128 Jun  9 02:09 16751
-rw-r--r-- 1 postgres postgres   2736128 Jun  9 02:09 16818
-rw-r--r-- 1 postgres postgres   2596864 Jun  8 10:34 17128
-rw-r--r-- 1 postgres postgres   2400256 Jun  9 01:39 16560
-rw-r--r-- 1 postgres postgres   2293760 Jun  9 01:49 16643
-rw-r--r-- 1 postgres postgres   2277376 Jun  9 02:09 16859
-rw-r--r-- 1 postgres postgres   2269184 Jun  9 02:09 16754
-rw-r--r-- 1 postgres postgres   2269184 Jun  9 02:09 16835
-rw-r--r-- 1 postgres postgres   2260992 Jun  9 01:55 16674
-rw-r--r-- 1 postgres postgres   2260992 Jun  9 01:55 16678
-rw-r--r-- 1 postgres postgres   2244608 Jun  9 02:09 16741
-rw-r--r-- 1 postgres postgres   2228224 Jun  9 02:09 16844
-rw-r--r-- 1 postgres postgres   2179072 Jun  9 02:09 16854
-rw-r--r-- 1 postgres postgres   2080768 Jun  9 02:04 16480
-rw-r--r-- 1 postgres postgres   2048000 Jun  9 02:04 16494
-rw-r--r-- 1 postgres postgres   2039808 Jun  9 02:04 16493
-rw-r--r-- 1 postgres postgres   2007040 Jun  9 02:05 16501
-rw-r--r-- 1 postgres postgres   1998848 Jun  8 10:35 17133
-rw-r--r-- 1 postgres postgres   1949696 Jun  9 02:14 16434
-rw-r--r-- 1 postgres postgres   1908736 Jun  9 01:55 16995
-rw-r--r-- 1 postgres postgres   1900544 Jun  9 01:55 16961
-rw-r--r-- 1 postgres postgres   1875968 Jun  9 01:56 16970
-rw-r--r-- 1 postgres postgres   1875968 Jun  9 01:55 16983
-rw-r--r-- 1 postgres postgres   1859584 Jun  9 01:55 16975
-rw-r--r-- 1 postgres postgres   1859584 Jun  9 01:56 16980
-rw-r--r-- 1 postgres postgres   1859584 Jun  9 01:55 16986
-rw-r--r-- 1 postgres postgres   1851392 Jun  9 01:54 16958
-rw-r--r-- 1 postgres postgres   1843200 Jun  9 01:55 16955
-rw-r--r-- 1 postgres postgres   1835008 Jun  9 01:55 17000
-rw-r--r-- 1 postgres postgres   1826816 Jun  9 01:49 16547
-rw-r--r-- 1 postgres postgres   1785856 Jun  9 01:49 16552
-rw-r--r-- 1 postgres postgres   1597440 Jun  9 01:56 16676
-rw-r--r-- 1 postgres postgres   1556480 Jun  9 01:54 16680
-rw-r--r-- 1 postgres postgres   1556480 Jun  9 01:56 16688
-rw-r--r-- 1 postgres postgres   1548288 Jun  9 01:35 16739
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16618
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16623
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16628
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16633
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16635
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16644
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16649
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 01:49 16655
-rw-r--r-- 1 postgres postgres   1540096 Jun  9 02:09 16840
-rw-r--r-- 1 postgres postgres   1515520 Jun  8 10:34 17106
-rw-r--r-- 1 postgres postgres   1515520 Jun  8 10:34 17126
-rw-r--r-- 1 postgres postgres    827392 Jun  9 01:35 16923
-rw-r--r-- 1 postgres postgres    811008 Jun  9 02:04 16486
-rw-r--r-- 1 postgres postgres    811008 Jun  9 02:04 16499
-rw-r--r-- 1 postgres postgres    811008 Jun  9 02:04 16504
-rw-r--r-- 1 postgres postgres    811008 Jun  9 02:04 16505
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16749
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16757
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16842
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16847
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16852
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16857
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16862
-rw-r--r-- 1 postgres postgres    786432 Jun  9 02:09 16867
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16964
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16966
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16968
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16973
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16978
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16989
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16990
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16993
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 16998
-rw-r--r-- 1 postgres postgres    786432 Jun  9 01:55 17003
-rw-r--r-- 1 postgres postgres    778240 Jun  9 01:49 16550
-rw-r--r-- 1 postgres postgres    778240 Jun  9 02:09 16744
-rw-r--r-- 1 postgres postgres    778240 Jun  9 02:09 16759
-rw-r--r-- 1 postgres postgres    753664 Jun  9 01:49 16555
-rw-r--r-- 1 postgres postgres    745472 Jun  8 10:34 17131
-rw-r--r-- 1 postgres postgres    745472 Jun  8 10:34 17136
-rw-r--r-- 1 postgres postgres    630784 Jun  2 17:04 1249
-rw-r--r-- 1 postgres postgres    614400 Jun  9 01:35 16713
-rw-r--r-- 1 postgres postgres    548864 Jun  9 01:55 16663
-rw-r--r-- 1 postgres postgres    532480 Jun  9 02:14 17033
-rw-r--r-- 1 postgres postgres    466944 Jun  2 17:04 2608
-rw-r--r-- 1 postgres postgres    409600 Apr  3 19:19 1255
-rw-r--r-- 1 postgres postgres    409600 Jun  2 17:04 2658
-rw-r--r-- 1 postgres postgres    368640 Jun  9 01:49 16524
-rw-r--r-- 1 postgres postgres    327680 Jun  9 02:05 16475
-rw-r--r-- 1 postgres postgres    319488 Jun  2 17:04 2674
-rw-r--r-- 1 postgres postgres    319488 Apr  3 19:19 2691
-rw-r--r-- 1 postgres postgres    303104 Jun  2 17:04 2673
-rw-r--r-- 1 postgres postgres    253952 Jun  9 01:34 16718
-rw-r--r-- 1 postgres postgres    245760 Jun  9 01:35 16729
-rw-r--r-- 1 postgres postgres    245760 Jun  9 01:34 16734
-rw-r--r-- 1 postgres postgres    229376 Jun  2 17:04 1247
-rw-r--r-- 1 postgres postgres    229376 Jun  9 01:49 17015
-rw-r--r-- 1 postgres postgres    221184 Jun  9 21:24 2619
-rw-r--r-- 1 postgres postgres    212992 Jun  9 02:04 1259
-rw-r--r-- 1 postgres postgres    204800 May 15 01:05 16508
-rw-r--r-- 1 postgres postgres    204800 Jun  9 01:49 16542
-rw-r--r-- 1 postgres postgres    196608 Jun  9 01:34 16912
-rw-r--r-- 1 postgres postgres    147456 May 15 03:09 16692
-rw-r--r-- 1 postgres postgres    147456 Apr  3 19:19 2838
-rw-r--r-- 1 postgres postgres    139264 Jun  9 01:34 16716
-rw-r--r-- 1 postgres postgres    139264 Jun  9 21:14 2610
-rw-r--r-- 1 postgres postgres    131072 Apr  3 19:19 2609
-rw-r--r-- 1 postgres postgres    114688 Jun  9 00:54 17041
-rw-r--r-- 1 postgres postgres    114688 Jun  2 17:04 2663
-rw-r--r-- 1 postgres postgres    114688 Apr  3 19:19 2675
-rw-r--r-- 1 postgres postgres    106496 Jun  9 01:39 17138
-rw-r--r-- 1 postgres postgres    106496 Apr  3 19:19 2617
-rw-r--r-- 1 postgres postgres    106496 Jun  2 17:04 2659
-rw-r--r-- 1 postgres postgres     98304 May 15 04:14 16700
-rw-r--r-- 1 postgres postgres     98304 Jun  2 17:04 2704
-rw-r--r-- 1 postgres postgres     93800 May 26 14:02 pg_internal.init
-rw-r--r-- 1 postgres postgres     90112 Jun  9 01:55 16666
-rw-r--r-- 1 postgres postgres     90112 May 15 03:09 16925
-rw-r--r-- 1 postgres postgres     90112 May 15 03:09 16930
-rw-r--r-- 1 postgres postgres     90112 May 15 03:09 17067
-rw-r--r-- 1 postgres postgres     90112 Apr  3 19:19 2689
-rw-r--r-- 1 postgres postgres     81920 Jun  9 02:14 17039
-rw-r--r-- 1 postgres postgres     81920 May 15 03:15 17072
-rw-r--r-- 1 postgres postgres     73728 Jun  9 01:49 16527
-rw-r--r-- 1 postgres postgres     65536 Apr  3 19:19 2618
-rw-r--r-- 1 postgres postgres     65536 Apr  3 19:19 2690
-rw-r--r-- 1 postgres postgres     57344 Jun  2 17:04 2606
-rw-r--r-- 1 postgres postgres     49152 Jun  9 02:04 16491
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:49 16545
-rw-r--r-- 1 postgres postgres     49152 May 15 03:09 16695
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:34 16721
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:34 16732
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:34 16737
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:35 16915
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:49 17005
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:49 17010
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:49 17021
-rw-r--r-- 1 postgres postgres     49152 Jun  9 01:49 17028
-rw-r--r-- 1 postgres postgres     49152 May 15 03:54 17054
-rw-r--r-- 1 postgres postgres     49152 Jun  8 10:45 17059
-rw-r--r-- 1 postgres postgres     49152 Jun  8 06:24 17090
-rw-r--r-- 1 postgres postgres     49152 Jun  8 09:59 17108
-rw-r--r-- 1 postgres postgres     49152 Jun  8 02:44 17113
-rw-r--r-- 1 postgres postgres     40960 Apr  3 19:19 11429
-rw-r--r-- 1 postgres postgres     40960 May 15 01:05 16514
-rw-r--r-- 1 postgres postgres     40960 Jun  9 01:49 16529
-rw-r--r-- 1 postgres postgres     40960 May 15 03:09 16938
-rw-r--r-- 1 postgres postgres     40960 May 15 03:10 16950
-rw-r--r-- 1 postgres postgres     40960 Jun  9 01:49 17026
-rw-r--r-- 1 postgres postgres     40960 Jun  9 00:54 17047
-rw-r--r-- 1 postgres postgres     40960 Jun  8 02:40 17118
-rw-r--r-- 1 postgres postgres     40960 Jun  7 02:24 17176
-rw-r--r-- 1 postgres postgres     40960 Jun  2 17:04 2664
-rw-r--r-- 1 postgres postgres     40960 Apr  3 19:19 2669
-rw-r--r-- 1 postgres postgres     40960 Jun  8 02:54 2696
-rw-r--r-- 1 postgres postgres     32768 Jun  9 01:49 16532
-rw-r--r-- 1 postgres postgres     32768 Jun  9 01:30 16591
-rw-r--r-- 1 postgres postgres     32768 May 15 04:14 16711
-rw-r--r-- 1 postgres postgres     32768 May 15 03:09 16928
-rw-r--r-- 1 postgres postgres     32768 May 15 03:09 16933
-rw-r--r-- 1 postgres postgres     32768 May 15 03:09 16935
-rw-r--r-- 1 postgres postgres     32768 May 15 03:10 16945
-rw-r--r-- 1 postgres postgres     32768 Jun  9 01:29 17049
-rw-r--r-- 1 postgres postgres     32768 May 15 03:54 17057
-rw-r--r-- 1 postgres postgres     32768 May 15 03:09 17070
-rw-r--r-- 1 postgres postgres     32768 May 15 03:15 17075
-rw-r--r-- 1 postgres postgres     32768 Jun  9 01:39 17141
-rw-r--r-- 1 postgres postgres     32768 Apr  3 19:19 2653
-rw-r--r-- 1 postgres postgres     32768 Jun  2 17:04 2662
-rw-r--r-- 1 postgres postgres     32768 Apr  3 19:19 2686
-rw-r--r-- 1 postgres postgres     32768 Apr  3 19:19 2688
-rw-r--r-- 1 postgres postgres     32768 Jun  2 17:04 2703
-rw-r--r-- 1 postgres postgres     32768 Apr  3 19:19 3609
-rw-r--r-- 1 postgres postgres     24576 Jun  9 01:30 16588
-rw-r--r-- 1 postgres postgres     24576 Apr  3 19:19 2602
-rw-r--r-- 1 postgres postgres     16384 May  6 03:15 16416
-rw-r--r-- 1 postgres postgres     16384 May  5 14:24 16418
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16571
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16579
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16581
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16586
-rw-r--r-- 1 postgres postgres     16384 Jun  8 10:45 16600
-rw-r--r-- 1 postgres postgres     16384 Jun  8 10:45 16603
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:30 16605
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:30 16606
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16878
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16883
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16891
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16892
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16894
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16903
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16904
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:35 16910
-rw-r--r-- 1 postgres postgres     16384 May 15 03:09 16939
-rw-r--r-- 1 postgres postgres     16384 May 15 03:09 16942
-rw-r--r-- 1 postgres postgres     16384 May 15 03:09 16948
-rw-r--r-- 1 postgres postgres     16384 May 15 03:09 16953
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:49 17008
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:49 17013
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:49 17024
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:49 17031
-rw-r--r-- 1 postgres postgres     16384 Jun  9 01:29 17052
-rw-r--r-- 1 postgres postgres     16384 Jun  8 10:45 17065
-rw-r--r-- 1 postgres postgres     16384 May 15 04:24 17080
-rw-r--r-- 1 postgres postgres     16384 Jun  9 02:14 17088
-rw-r--r-- 1 postgres postgres     16384 Jun  8 06:24 17096
-rw-r--r-- 1 postgres postgres     16384 Jun  4 04:29 17101
-rw-r--r-- 1 postgres postgres     16384 Jun  8 09:59 17111
-rw-r--r-- 1 postgres postgres     16384 Jun  8 02:44 17116
-rw-r--r-- 1 postgres postgres     16384 Jun  8 02:44 17121
-rw-r--r-- 1 postgres postgres     16384 Jun  8 11:04 17146
-rw-r--r-- 1 postgres postgres     16384 Jun  8 11:04 17151
-rw-r--r-- 1 postgres postgres     16384 Apr 29 17:34 17156
-rw-r--r-- 1 postgres postgres     16384 Apr 29 17:34 17161
-rw-r--r-- 1 postgres postgres     16384 Jun  8 02:25 17169
-rw-r--r-- 1 postgres postgres     16384 Jun  6 08:35 17174
-rw-r--r-- 1 postgres postgres     16384 Jun  7 02:24 17182
-rw-r--r-- 1 postgres postgres     16384 Jun  8 09:55 17187
-rw-r--r-- 1 postgres postgres     16384 May 19 02:39 17192
-rw-r--r-- 1 postgres postgres     16384 May 13 05:19 17197
-rw-r--r-- 1 postgres postgres     16384 Jun  8 09:55 17202
-rw-r--r-- 1 postgres postgres     16384 May  5 14:24 17210
-rw-r--r-- 1 postgres postgres     16384 May  5 14:24 17218
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17238
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17240
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17242
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17247
-rw-r--r-- 1 postgres postgres     16384 May  6 17:04 17255
-rw-r--r-- 1 postgres postgres     16384 May  6 17:04 17257
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17262
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17267
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17272
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17292
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17294
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17296
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17304
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17305
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17311
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17316
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17324
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17332
-rw-r--r-- 1 postgres postgres     16384 May  5 14:25 17334
-rw-r--r-- 1 postgres postgres     16384 May  5 23:44 17339
-rw-r--r-- 1 postgres postgres     16384 May  6 00:35 17344
-rw-r--r-- 1 postgres postgres     16384 May  6 00:35 17349
-rw-r--r-- 1 postgres postgres     16384 May  6 00:35 17354
-rw-r--r-- 1 postgres postgres     16384 May  6 02:00 17359
-rw-r--r-- 1 postgres postgres     16384 May  6 03:20 17364
-rw-r--r-- 1 postgres postgres     16384 May  6 18:39 17372
-rw-r--r-- 1 postgres postgres     16384 May 16 09:54 17794
-rw-r--r-- 1 postgres postgres     16384 May 15 08:25 17799
-rw-r--r-- 1 postgres postgres     16384 May 26 14:10 17807
-rw-r--r-- 1 postgres postgres     16384 May 26 14:19 17815
-rw-r--r-- 1 postgres postgres     16384 Jun  2 17:04 18302
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2603
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2605
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2607
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2616
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2650
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2651
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2652
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2654
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2655
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2660
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2661
-rw-r--r-- 1 postgres postgres     16384 Jun  2 17:04 2665
-rw-r--r-- 1 postgres postgres     16384 Jun  2 17:04 2666
-rw-r--r-- 1 postgres postgres     16384 Jun  2 17:04 2667
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2668
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2670
-rw-r--r-- 1 postgres postgres     16384 Jun  2 17:04 2678
-rw-r--r-- 1 postgres postgres     16384 Jun  2 17:04 2679
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2681
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2682
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2684
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2685
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2687
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2692
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2693
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2699
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2701
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2702
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2754
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2755
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2756
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2757
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 2839
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3603
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3604
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3605
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3606
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3607
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3608
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3712
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3766
-rw-r--r-- 1 postgres postgres     16384 Apr  3 19:19 3767
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11433
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11434
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11438
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11439
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11443
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11444
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11448
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11449
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11453
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11454
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11458
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 11463
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 1250
-rw-r--r-- 1 postgres postgres      8192 Apr 28 19:58 16396
-rw-r--r-- 1 postgres postgres      8192 Apr 28 19:58 16397
-rw-r--r-- 1 postgres postgres      8192 May  6 03:15 16404
-rw-r--r-- 1 postgres postgres      8192 Apr 28 19:58 16409
-rw-r--r-- 1 postgres postgres      8192 May  5 14:24 16410
-rw-r--r-- 1 postgres postgres      8192 Apr 28 19:58 16415
-rw-r--r-- 1 postgres postgres      8192 Apr 28 19:59 16425
-rw-r--r-- 1 postgres postgres      8192 Apr 28 19:59 16431
-rw-r--r-- 1 postgres postgres      8192 Apr 28 21:00 16461
-rw-r--r-- 1 postgres postgres      8192 Apr 28 21:00 16462
-rw-r--r-- 1 postgres postgres      8192 Apr 28 21:00 16485
-rw-r--r-- 1 postgres postgres      8192 Apr 28 21:00 16488
-rw-r--r-- 1 postgres postgres      8192 Apr 28 21:50 16513
-rw-r--r-- 1 postgres postgres      8192 Apr 28 21:50 16521
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:15 16539
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16562
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:15 16567
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16568
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16573
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:15 16578
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16583
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:15 16597
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:15 16599
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:15 16614
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:22 16705
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:22 16708
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:22 16728
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:26 16766
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16869
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:29 16874
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16875
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16880
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16885
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16886
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16897
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16900
-rw-r--r-- 1 postgres postgres      8192 Jun  9 01:35 16907
-rw-r--r-- 1 postgres postgres      8192 Apr 28 23:32 16922
-rw-r--r-- 1 postgres postgres      8192 Apr 29 00:20 17020
-rw-r--r-- 1 postgres postgres      8192 Apr 29 00:30 17038
-rw-r--r-- 1 postgres postgres      8192 Apr 29 00:30 17046
-rw-r--r-- 1 postgres postgres      8192 Apr 29 00:39 17064
-rw-r--r-- 1 postgres postgres      8192 May 15 04:24 17077
-rw-r--r-- 1 postgres postgres      8192 Jun  9 02:14 17082
-rw-r--r-- 1 postgres postgres      8192 Apr 29 02:38 17087
-rw-r--r-- 1 postgres postgres      8192 Apr 29 02:38 17095
-rw-r--r-- 1 postgres postgres      8192 Jun  4 04:29 17098
-rw-r--r-- 1 postgres postgres      8192 Jun  8 11:04 17143
-rw-r--r-- 1 postgres postgres      8192 Jun  8 11:04 17148
-rw-r--r-- 1 postgres postgres      8192 Apr 29 17:34 17153
-rw-r--r-- 1 postgres postgres      8192 Apr 29 17:34 17158
-rw-r--r-- 1 postgres postgres      8192 Jun  8 02:25 17163
-rw-r--r-- 1 postgres postgres      8192 Apr 29 23:38 17168
-rw-r--r-- 1 postgres postgres      8192 Jun  6 08:35 17171
-rw-r--r-- 1 postgres postgres      8192 May  1 23:18 17181
-rw-r--r-- 1 postgres postgres      8192 Jun  8 09:55 17184
-rw-r--r-- 1 postgres postgres      8192 May 19 02:39 17189
-rw-r--r-- 1 postgres postgres      8192 May 13 05:19 17194
-rw-r--r-- 1 postgres postgres      8192 Jun  8 09:55 17199
-rw-r--r-- 1 postgres postgres      8192 May  5 14:24 17204
-rw-r--r-- 1 postgres postgres      8192 May  5 14:22 17209
-rw-r--r-- 1 postgres postgres      8192 May  5 14:24 17212
-rw-r--r-- 1 postgres postgres      8192 May  5 14:22 17217
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17220
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17221
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17223
-rw-r--r-- 1 postgres postgres      8192 May  5 14:22 17231
-rw-r--r-- 1 postgres postgres      8192 May  5 14:22 17234
-rw-r--r-- 1 postgres postgres      8192 May  5 14:22 17237
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17244
-rw-r--r-- 1 postgres postgres      8192 May  6 17:04 17249
-rw-r--r-- 1 postgres postgres      8192 May  6 17:04 17252
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17259
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17264
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17269
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17274
-rw-r--r-- 1 postgres postgres      8192 May  5 14:23 17279
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17280
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17283
-rw-r--r-- 1 postgres postgres      8192 May  5 14:23 17288
-rw-r--r-- 1 postgres postgres      8192 May  5 14:23 17291
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17298
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17299
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17308
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17313
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17318
-rw-r--r-- 1 postgres postgres      8192 May  5 14:23 17323
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17326
-rw-r--r-- 1 postgres postgres      8192 May  5 14:25 17327
-rw-r--r-- 1 postgres postgres      8192 May  5 23:44 17336
-rw-r--r-- 1 postgres postgres      8192 May  6 00:35 17341
-rw-r--r-- 1 postgres postgres      8192 May  6 00:35 17346
-rw-r--r-- 1 postgres postgres      8192 May  6 00:35 17351
-rw-r--r-- 1 postgres postgres      8192 May  6 02:00 17356
-rw-r--r-- 1 postgres postgres      8192 May  6 03:20 17361
-rw-r--r-- 1 postgres postgres      8192 May  6 18:39 17366
-rw-r--r-- 1 postgres postgres      8192 May  6 18:35 17371
-rw-r--r-- 1 postgres postgres      8192 May 16 09:54 17791
-rw-r--r-- 1 postgres postgres      8192 May 15 08:25 17796
-rw-r--r-- 1 postgres postgres      8192 May 26 14:10 17801
-rw-r--r-- 1 postgres postgres      8192 May 26 14:00 17806
-rw-r--r-- 1 postgres postgres      8192 May 26 14:19 17809
-rw-r--r-- 1 postgres postgres      8192 May 26 14:07 17814
-rw-r--r-- 1 postgres postgres      8192 Jun  2 17:04 18299
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2600
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2601
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2612
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2615
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2620
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2656
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2657
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2680
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2683
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2753
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2831
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2833
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2835
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2837
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 2841
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 3502
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 3503
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 3600
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 3601
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 3602
-rw-r--r-- 1 postgres postgres      8192 Apr  3 19:19 3764
-rw-r--r-- 1 postgres postgres         4 Apr  3 19:19 PG_VERSION
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11431
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11436
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11441
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11446
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11451
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11456
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11459
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 11461
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 1248
-rw-r--r-- 1 postgres postgres         0 Apr 28 19:58 16392
-rw-r--r-- 1 postgres postgres         0 Apr 28 19:58 16394
-rw-r--r-- 1 postgres postgres         0 Apr 28 19:58 16407
-rw-r--r-- 1 postgres postgres         0 Apr 28 19:58 16413
-rw-r--r-- 1 postgres postgres         0 Apr 28 19:59 16423
-rw-r--r-- 1 postgres postgres         0 Apr 28 19:59 16429
-rw-r--r-- 1 postgres postgres         0 Apr 28 21:00 16456
-rw-r--r-- 1 postgres postgres         0 Apr 28 21:00 16459
-rw-r--r-- 1 postgres postgres         0 Apr 28 21:00 16478
-rw-r--r-- 1 postgres postgres         0 Apr 28 21:00 16483
-rw-r--r-- 1 postgres postgres         0 Apr 28 21:50 16511
-rw-r--r-- 1 postgres postgres         0 Apr 28 21:50 16519
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:15 16537
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:15 16565
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:15 16576
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:15 16592
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:15 16595
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:15 16612
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:22 16703
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:22 16706
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:22 16726
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:26 16764
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:29 16872
-rw-r--r-- 1 postgres postgres         0 Apr 28 23:32 16920
-rw-r--r-- 1 postgres postgres         0 Apr 29 00:20 17018
-rw-r--r-- 1 postgres postgres         0 Apr 29 00:30 17036
-rw-r--r-- 1 postgres postgres         0 Apr 29 00:30 17044
-rw-r--r-- 1 postgres postgres         0 Apr 29 00:39 17062
-rw-r--r-- 1 postgres postgres         0 Apr 29 02:38 17085
-rw-r--r-- 1 postgres postgres         0 Apr 29 02:38 17093
-rw-r--r-- 1 postgres postgres         0 Apr 29 23:38 17166
-rw-r--r-- 1 postgres postgres         0 May  1 23:18 17179
-rw-r--r-- 1 postgres postgres         0 May  5 14:22 17207
-rw-r--r-- 1 postgres postgres         0 May  5 14:22 17215
-rw-r--r-- 1 postgres postgres         0 May  5 14:22 17229
-rw-r--r-- 1 postgres postgres         0 May  5 14:22 17232
-rw-r--r-- 1 postgres postgres         0 May  5 14:22 17235
-rw-r--r-- 1 postgres postgres         0 May  5 14:23 17277
-rw-r--r-- 1 postgres postgres         0 May  5 14:23 17284
-rw-r--r-- 1 postgres postgres         0 May  5 14:23 17289
-rw-r--r-- 1 postgres postgres         0 May  5 14:23 17321
-rw-r--r-- 1 postgres postgres         0 May  6 18:35 17369
-rw-r--r-- 1 postgres postgres         0 May 26 14:00 17804
-rw-r--r-- 1 postgres postgres         0 May 26 14:07 17812
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2604
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2611
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2613
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2614
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2830
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2832
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2834
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2836
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 2840
-rw-r--r-- 1 postgres postgres         0 Apr  3 19:19 3501

The reply_raw file (16386) is 917,782,528 bytes = 112,034 pages (1 page = 8192 bytes). For comparison, the stats tables (above) reported 105k pages, but that was over 12 hours ago so these are probably consistent.

April 21, 2009

hub25m Migration

Filed under: Uncategorized — sdssops @ 3:40 pm

Environment

Use the following to get ’setup’ and ’sdss3install’ working:

source ~sdss3/products/eups/bin/setups.sh
setup sdss3tools

Update Archiver

First, tag the archiver:

% svn copy svn+ssh://sdss3svn@sdss3.org/repo/ops/archiver/trunk svn+ssh://sdss3svn@sdss3.org/repo/ops/archiver/tags/v1_0 -m 'Initial tag for testing at APO'

Committed revision 98917.

Next, install the tag on hub25m:

% sdss3install -c  ops/archiver v1_0
4:     archiver-v1_0.script -c
A    /home/dkirkby/archiver-v1_0
A    /home/dkirkby/archiver-v1_0/python
A    /home/dkirkby/archiver-v1_0/python/archiver
A    /home/dkirkby/archiver-v1_0/python/archiver/archiver.ini
A    /home/dkirkby/archiver-v1_0/python/archiver/actors.py
A    /home/dkirkby/archiver-v1_0/python/archiver/__init__.py
A    /home/dkirkby/archiver-v1_0/python/archiver/expression.py
A    /home/dkirkby/archiver-v1_0/python/archiver/protocol.py
A    /home/dkirkby/archiver-v1_0/python/archiver/database.py
A    /home/dkirkby/archiver-v1_0/python/archiver/server.py
A    /home/dkirkby/archiver-v1_0/setup.py
A    /home/dkirkby/archiver-v1_0/ups
A    /home/dkirkby/archiver-v1_0/ups/archiver.table
A    /home/dkirkby/archiver-v1_0/examples
A    /home/dkirkby/archiver-v1_0/examples/benchmark.py
A    /home/dkirkby/archiver-v1_0/examples/trace.cc
A    /home/dkirkby/archiver-v1_0/examples/config.ini
A    /home/dkirkby/archiver-v1_0/examples/logclient.py
Exported revision 98925.
WARNING: Using table file /home/dkirkby/archiver-v1_0/ups/archiver.table
Setting up:     archiver             Flavor: Linux64    Version: LOCAL:/home/dkirkby/archiver-v1_0
Setting up:      opscore             Flavor: Linux64    Version: 1.0m1
Setting up:       external           Flavor: Linux64    Version: 1.0m1
Setting up:       actorkeys          Flavor: Linux64    Version: 1.0
kargs= {'description': 'Data archiver for SDSS-III', 'name': 'archiver', 'scripts': [], 'data_files': [('ups', ['ups/archiver.table'])], 'packages': ['archiver'], 'package_dir': {'archiver': 'python/archiver'}}
running build
running build_py
creating build
creating build/lib
creating build/lib/archiver
copying python/archiver/server.py -> build/lib/archiver
copying python/archiver/protocol.py -> build/lib/archiver
copying python/archiver/expression.py -> build/lib/archiver
copying python/archiver/actors.py -> build/lib/archiver
copying python/archiver/__init__.py -> build/lib/archiver
copying python/archiver/database.py -> build/lib/archiver
kargs= {'description': 'Data archiver for SDSS-III', 'name': 'archiver', 'scripts': [], 'data_files': [('ups', ['ups/archiver.table'])], 'packages': ['archiver'], 'package_dir': {'archiver': 'python/archiver'}}
running install
running build
running build_py
running install_lib
creating /home/sdss3/products/Linux64/archiver/v1_0/python
creating /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
copying build/lib/archiver/server.py -> /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
copying build/lib/archiver/protocol.py -> /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
copying build/lib/archiver/expression.py -> /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
copying build/lib/archiver/actors.py -> /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
copying build/lib/archiver/__init__.py -> /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
copying build/lib/archiver/database.py -> /home/sdss3/products/Linux64/archiver/v1_0/python/archiver
byte-compiling /home/sdss3/products/Linux64/archiver/v1_0/python/archiver/server.py to server.pyc
byte-compiling /home/sdss3/products/Linux64/archiver/v1_0/python/archiver/protocol.py to protocol.pyc
byte-compiling /home/sdss3/products/Linux64/archiver/v1_0/python/archiver/expression.py to expression.pyc
byte-compiling /home/sdss3/products/Linux64/archiver/v1_0/python/archiver/actors.py to actors.pyc
byte-compiling /home/sdss3/products/Linux64/archiver/v1_0/python/archiver/__init__.py to __init__.pyc
byte-compiling /home/sdss3/products/Linux64/archiver/v1_0/python/archiver/database.py to database.pyc
running install_data
creating /home/sdss3/products/Linux64/archiver/v1_0/ups
copying ups/archiver.table -> /home/sdss3/products/Linux64/archiver/v1_0/ups
running install_egg_info
Writing /home/sdss3/products/Linux64/archiver/v1_0/python/archiver-0.0.0-py2.5.egg-info
INFO: Guessing product = archiver; version = v1_0
INFO : Root = "/home/sdss3/products/Linux64/archiver/v1_0"
INFO : creating version file /home/sdss3/products/ups_db/archiver/v1_0.version for archiver v1_0 (flavor: Linux64, db: /home/sdss3/products)
INFO : Modifying current.chain for archiver v1_0 (flavor: Linux64)

The end result is that the new tag is installed in /home/sdss3/products/Linux64/archiver/v1_0/ and my home dir has a new file archiver-v1_0.script (I think this is what just ran… do I need to keep it around?)

None of the ops stuff is on the PYTHONPATH yet. Use setup to fix this:

[dkirkby@hub25m ~]$ echo $PYTHONPATH
/home/sdss3/products/Linux64/sdss3tools/1.0/python:
/home/sdss3/products/eups/bin

[dkirkby@hub25m ~]$ setup archiver

[dkirkby@hub25m ~]$ echo $PYTHONPATH
/home/sdss3/products/Linux64/archiver/v1_0/python:
/home/sdss3/products/Linux64/opscore/1.0m1/python:
/home/sdss3/products/Linux64/actorkeys/1.0/python:
/home/sdss3/products/Linux64/external/1.0m1/python:
/home/sdss3/products/Linux64/sdss3tools/1.0/python:
/home/sdss3/products/eups/bin

Note how setting up the archiver causes all of its dependencies to also be setup.

Running Archiver

Try running using -m (module) option to python:

[dkirkby@hub25m ~]$ python -m archiver.server --help
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/runpy.py", line 95, in run_module
    filename, loader, alter_sys)
  File "/usr/local/lib/python2.5/runpy.py", line 52, in _run_module_code
    mod_name, mod_fname, mod_loader)
  File "/usr/local/lib/python2.5/runpy.py", line 32, in _run_code
    exec code in run_globals
  File "/home/sdss3/products/Linux64/archiver/v1_0/python/archiver/server.py", line 88, in <module>
    import opscore.utility.config as config
ImportError: No module named config

Fails because I still need to update dependent packages.

Use Updated sdss3tools

Russell has committed a new version of sdss3tools that will copy a package’s etc/ directory and allows additional data_dirs to be specified (I need to add web/).

How do I get this working on hub25m? Look at what’s there already:

[dkirkby@hub25m ~]$ setup -v -l sdss3tools
   1.0       	/home/sdss3/products	/home/sdss3/products/Linux64/sdss3tools/1.0		 Current Setup
   trunk     	/home/sdss3/products	/home/sdss3/products/Linux64/sdss3tools/trunk

I want to update the trunk version without changing the current setup. Try this:

[dkirkby@hub25m ~]$ sdss3install -F -v sdss3tools trunk

Seems to work. Now use it via:

[dkirkby@hub25m ~]$ setup -v sdss3tools trunk
Setting up:     sdss3tools           Flavor: Linux64    Version: trunk

April 4, 2009

Archiver – Hub Integration

Filed under: Uncategorized — sdssops @ 1:43 am

Archiver – Hub Integration

Baseline

First, check that the archiver is working as before after all the changes to opscore. Currently using svn revision 98723. Run the server and feed it with TCC log files:

% python server.py
% python logclient.py --ival=1000 --burst=10 --repeat=10 --trace=timing.dat tcc-log/2008-12-01T19:00:02.log
% cp /tmp/archiver-69547/reply_hdr.trace .

Run the trace.cc macro in ROOT to create the following plots:

rateselapsed

Connecting to the Hub

I can see the stream of all reply messages using:

% telnet hub25m.apo.nmsu.edu 6095
Trying 192.41.211.191...
Connected to hub25m.apo.nmsu.edu.
Escape character is '^]'.
.hub 0 hub i user="APO.Craig","TUI","2009-02-05 1.6a6","Darwin-9.6.0-i386-32bit aqua","192.41.211.239","dhcp-239.apo.nmsu.edu"
.hub 0 hub i Commanders="APO.Craig","nclient_5","client-7","nclient_10"
.hub 0 hub i users="APO.Craig"
(etc)

I currently configure the reactor to accept connections using:

reactor.listenTCP(1966,factory)
reactor.listenUNIX('/tmp/archiver.sock',factory)

For the hub, I need to connect instead of listen. Try:

client = protocol.ClientCreator(reactor,archiver.protocol.ReplyReceiver)
client.connectTCP("hub25m.apo.nmsu.edu",6095)

April 3, 2009

EPICS at APO

Filed under: apo — sdssops @ 11:49 pm

EPICS and the TPM at APO

Establishing the Environment

From sdsshost2:

% source /usr/local/etc/setups.csh
% setup epics
% setup tmp

Versions in Production

Jon says it is R3.13.5. Here are official release notes (Sep 2001).

Latest version is R3.14.10.

Most of what I need is in the Channel Access Reference Manual: R3.14 (more complete), or R3.12 (skimpy, also used in R3.13). Page 14 of this tutorial lists the 3.14-3.12 differences.

Listening to Record Updates

Try the ca_test program in $EPICS_BASE/bin/Linux/ (already in path) documented here for R3.14. Its very verbose so caget or camonitor are probably more useful.

Use the following weather records for testing, which have changing values that are meaningful during the day:

record(waveAnl,"tpm_wspeed_W") {
		field(DESC, "WEATHER:Wind speed - waveform")
		field(INP, "tpm_wspeed_C")
		field(EGUX, "mph")
		field(NELM, "50")
		field(FLNK, "tpm_wspeed_M")}
record(waveAnl,"tpm_wdir_W") {
		field(DESC, "WEATHER:Wind direction - waveform")
		field(INP, "tpm_wdir_C")
		field(EGUX, "deg")
		field(NELM, "50")
		field(FLNK, "tpm_wdir_M")}

These are actually “waveforms” that store 5 values. Use the “.MEAN” attribute to get their mean:

...R3.13.5_SDSS/base % caget tpm_wspeed_W
tpm_wspeed_W 5 4.51112 4.51112 4.12928 4.12928 4.12928
...R3.13.5_SDSS/base % caget tpm_wspeed_W.MEAN
tpm_wspeed_W.MEAN             4.06237
...R3.13.5_SDSS/base % caget tpm_wdir_W
tpm_wdir_W 5 100.657 100.415 100.261 96.8774 96.8774
...R3.13.5_SDSS/base % caget tpm_wdir_W.MEAN
tpm_wdir_W.MEAN               339.097

What are the TPM Records?

Count how many records are defined in each file in $TPM_DIR/db/

% fgrep -c "record" *.db
ab.db:1
absub.db:206
ampstat.db:5
axis.db:90
dmig_dt.db:10
encl_humid.db:8
galil.db:8
galil_m1.db:18
galil_m2.db:22
humid.db:7
ilock.db:13
imager_ups.db:7
ip470.db:12
mig.db:24
plc_analog.db:12
pmss.db:36
scale.db:8
scaled.db:3
slip.db:1
temp_alarms.db:26
temp_means.db:27
tests_axis.db:6
tmicro_block.db:48
tmicro.db:15
tpmAlarms.db:0
util2.db:4
util.db:18
waveAnl.db:24

To open the TPM epics displays use:

% setup tpm
% dm tpm.dl &

This is what it looks like:

tpm

Developing a Bridge to the Archiver

This tutorial talks about writing your own CA client. Here is another similar tutorial.

Copy the simplecaget.c from the first tutorial. Some of the calls are R3.14, so replace these with their R3.13 equivalents. Compile and link with:

% gcc -I$EPICS_BASE/include -c catest.c
% gcc -o catest -L$EPICS_BASE/lib/Linux/ catest.o -lca -lCom -lm

Test the program:

% ./catest tpm_wdir_W.MEAN
The value of tpm_wdir_W.MEAN is 348
% caget tpm_wdir_W.MEAN
tpm_wdir_W.MEAN               341.068

The program asks for the value as a STRING, so the truncation is expected.

Instead of reading a single value, I want to subscribe to a channel and get all updates. Look at the R3.13 ca_add_event() docs…the handler is passed one of these:

/*
 * Arguments passed to event handlers and get/put call back handlers.
 *
 * The status field below is the CA ECA_XXX status of the requested
 * operation which is saved from when the operation was attempted in the
 * server and copied back to the clients call back routine.
 * If the status is not ECA_NORMAL then the dbr pointer will be NULL
 * and the requested operation can not be assumed to be successful.
 */
typedef struct	event_handler_args{
	void		*usr;	/* User argument supplied when event added 	*/
	struct channel_in_use
			*chid;	/* Channel id					*/
	long		type;	/* the type of the value returned		*/
	long		count;	/* the element count of the item returned	*/
	READONLY void	*dbr;	/* Pointer to the value returned		*/
	int		status;	/* ECA_XXX Status of the requested op from server	*/
}evargs;

December 4, 2008

SQL Tests

Filed under: python, sql — sdssops @ 1:40 am

MySQL Server

Start server with:

sudo /usr/local/mysql/bin/mysqld_safe
^Z
bg

Secure initial accounts with:

% mysql -u root
mysql> select host,user,password,select_priv,insert_priv,create_priv from mysql.user;
+------------------------+------+----------+-------------+-------------+-------------+
| host                   | user | password | select_priv | insert_priv | create_priv |
+------------------------+------+----------+-------------+-------------+-------------+
| localhost              | root |          | Y           | Y           | Y           |
| hep-dhcp167.ps.uci.edu | root |          | Y           | Y           | Y           |
| 127.0.0.1              | root |          | Y           | Y           | Y           |
| localhost              |      |          | N           | N           | N           |
| hep-dhcp167.ps.uci.edu |      |          | N           | N           | N           |
+------------------------+------+----------+-------------+-------------+-------------+
mysql> delete from mysql.user where user='';
Query OK, 2 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
mysql> SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('***');
(for all 3 hosts)
mysql> select host,user,password,select_priv,insert_priv,create_priv from mysql.user;
+------------------------+------+-------------------------------------------+-------------+-------------+-------------+
| host                   | user | password                                  | select_priv | insert_priv | create_priv |
+------------------------+------+-------------------------------------------+-------------+-------------+-------------+
| localhost              | root | *D2949FF4D5C267865367B8551C09AAE2EFF95CA6 | Y           | Y           | Y           |
| hep-dhcp167.ps.uci.edu | root | *D2949FF4D5C267865367B8551C09AAE2EFF95CA6 | Y           | Y           | Y           |
| 127.0.0.1              | root | *D2949FF4D5C267865367B8551C09AAE2EFF95CA6 | Y           | Y           | Y           |
+------------------------+------+-------------------------------------------+-------------+-------------+-------------+
mysql> FLUSH PRIVILEGES;

Create a new testing account following these instructions:

 % mysql -u root -p
***
mysql> create user 'testing' identified by '***';
mysql> grant all on test.* to 'testing';
mysql> FLUSH PRIVILEGES;

Insert 1,000,000 rows into empty table:

every 1   309.393u 80.625s 19:37.77 33.1% 0+0k 0+2io 0pf+0w  elapsed = 1177.52  rate = 0.85 kHz
every 10  221.926u 33.224s 7:41.58 55.2% 0+0k 0+50io 0pf+0w  elapsed =  461.21  rate = 2.17 kHz
every 100 192.134u 25.665s 5:53.88 61.5% 0+0k 1+52io 0pf+0w  elapsed =  353.23  rate = 2.83 kHz
no trans  140.489u 18.664s 4:54.63 54.0% 0+0k 0+53io 0pf+0w  elapsed =  294.21  rate = 3.40 kHz
blackhole 139.008u 18.392s 3:43.95 70.2% 0+0k 0+48io 0pf+0w  elapsed =  223.70  rate = 4.47 kHz

PostgreSQL Server

Start/stop server with:

sudo -u postgres /Library/PostgreSQL/8.3/bin/pg_ctl start -w -D /Library/PostgreSQL/8.3/data -l /tmp/postgres.log
sudo -u postgres /Library/PostgreSQL/8.3/bin/pg_ctl stop -D /Library/PostgreSQL/8.3/data

Create a new test database and testing account with:

postgres=# create role testing password '***';
postgres=# create database test owner testing;
postgres-# grant all on database test to testing;
postgres=# alter role testing LOGIN;
postgres=# alter role testing SUPERUSER;

The ‘SUPERUSER’ privilege is necessary for ‘COPY <table> FROM <file>…’

October 7, 2008

Hub Design

Filed under: Uncategorized — sdssops @ 11:15 pm

What is in tron right now?

  • CPL/ 780 lines
  • Client/ 2318 lines
  • Hub/ 3328 lines
  • IO/ 1158 lines
  • Misc/ 5676 lines (mostly FITS stuff)
  • Nubs/ 1603 lines
  • Parsing/ 365
  • TclActor/ 1630 lines
  • Vocab/ 1907 lines
  • clients/ 16467
    • expose/ 4211
    • fits/ 3685
    • guiders/ 6352
    • telmech/ 1419

October 1, 2008

Automatic Parsing of SOP

Filed under: Uncategorized — sdssops @ 12:57 am

Goal is to automatically build a cross-referenced index of SOP procedures from the tcl code base. In order to do this, I need to be able to parse a tcl file and identify ‘proc’ commands (which define a new procedure) and generic words that might be procedure names and thus invoke a defined procedure.

The tcl basic syntax is described here. There has already been a long discussion about tcl parsers concluding that it is not really possible. Since my goal is mostly lexical analysis with very simple parsing requirements, I still think this is a reasonable thing to try.

PLY is a pure-python implementation of lex/yacc that is actively maintained and has a nice design philosophy. Try that. Works great, but the tcl ‘grammar’ is not easily expressed in BNF, so I am just using the lexer. What I need is actually simple enough that I could probably implement it without PLY at all, to avoid adding a new dependency…

Steph says that the lastest version of IOP is v4_39_0 (my local copy is v4_38_0) and sent me the following instructions for checking it out on sdsshost2:

umask 002
setup sdsscvs
cvs co iop

However, the ’setup…’ command prompts me for the root psswd, so that doesn’t work. Try and find an already checked out copy … ~observer/iop/ looks promising. Where was it checked out from?

[dkirkby@sdsshost2 ~observer/iop]$ cat ~observer/iop/CVS/Root
cvsuser@sdss.fnal.gov:/cvs/sdss

If I use this for my $CVSROOT, a cvs checkout fails, not surprisingly:

[dkirkby@sdsshost2 ~observer/iop]$ cvs co iop
sdssprd1.fnal.gov: No route to host
trying normal rsh (/usr/bin/rsh)  WARNING: NO ENCRYPTION!
sdssprd1.fnal.gov: No route to host
cvs [checkout aborted]: end of file from server (consult above messages if any)

Email Steph again and ask if ~observer/iop/ is v4_39_0…

I found what looks like a syntax error in v4_38 that I have fixed by hand:

% diff camera.tcl camera.tcl.orig
3024c3024
< 	return [cameraMatchResponse [cameraSend STARTFILLSEQ] "AUTOFILL VALVE OPEN F  \n" ]
---
> 	return [cameraMatchResponse [cameraSend STARTFILLSEQ]] "AUTOFILL VALVE OPEN F  \n" ]

Steph checked out the latest iop code into /userdev/observer/iop/ and I have grabbed that now. The syntax error is still there on line 3024 in etc/camera.tcl. This is the original:

proc startfillseq {} {
	return [cameraMatchResponse [cameraSend STARTFILLSEQ]] "AUTOFILL VALVE OPEN F  \n" ]
}

This looks like another, more subtle, error in an “if” command starting on line 89 of etc/camera.tcl:

if {[string range $im1 0 1] == "sp"  && [string range $im2 0 1] == "sp" } {
   if {[lindex $name1 1] != [lindex $name2 1]} {
   	error "Instruments $im1 and $im2 have different versions: $name1 $name2"
   }
} else if {$name1 != $name2} {
   error "Instruments $im1 and $im2 have different versions: $name1 $name2"
}

The “else if …” should either be written as “elseif …” or “else { if … }”. Check this with tclsh8.4:

% if 0 { } else if 1 { puts "hello" }
wrong # args: no expression after "if" argument
% if 0 { } elseif 1 { puts "hello" }
hello
% if 0 { } else { if 1 { puts "hello" } }
hello

I am getting a lot of “errors” due to EOL embedded within a command substitution [...]. For example, starting on line 525 of etc/astrom.tcl:

    set opts [list \
	    [list updateAstromData "update astrom data array"]\
	    [list \[skippyFrequency\] INTEGER 50 skippyFrequency "how often (in frames) should skippy be run"]\
	    [list \[ltMatchFrequency\] INTEGER 10 ltMatchFrequency "how often (in fields) should ltMatch be run"]\
	    [list \[flavors\] STRING [list science engineering] flavors "a tcl list of flavors on when astrom checks should be run"]
    ]

Test what is really allowed in tclsh8.4:

% puts [expr 1; expr 2]
2
% puts [expr 1
expr 2]
2

So multiple commands are allowed, and EOL is a valid command separator within [...]. So why did the author of the code above bother with all those escaped newlines?? Fix my parser accordingly…ok.

Here’s another possible syntax error:

   echo "endNight: Did all the sdssid?-1 drives make it through the night? \[y]:"

Shouldn’t the closing ] also be escaped in this string? I guess this isn’t required:

% puts "hello \[world\]"
hello [world]
% puts "hello [world]"
invalid command name "world"
% puts "hello \[world]"
hello [world]

What about unmatched braces within a Quoted?

% puts "hello {world}"
hello {world}
% puts "hello \{world}"
hello {world}
% puts "hello {{world}"
hello {{world}

It looks like I shouldn’t be parsing groups within a Quoted and should treat ‘}’ and ‘]’ like any other tokens. Here is a good test case:

% puts "{ \[ [expr 1+1] ] \}"
{ [ 2 ] }

Fixed parser … ok. Now this looks like a legitimate error on line 2704 of etc/highCommand.tcl:

   if { [ catch {checkVmeState asmGrp astro isbad} msg] } {
	error "Aborting: pool not set correctly, $msg
   }

The string is missing its closing quote. Here’s another, on line 285 of etc/iopStartup.tcl:

   regexp {^([0-9]+):?([0-9]+)} $listen foo listen ntry]

Shouldn’t be a closing ‘]’ ? Check this in tclsh8.4 … it looks like an unmatched ] or } is ok in commands, just like in a Quoted:

% puts \{hello}
{hello}
% puts \[hello]
[hello]

So in this case, the extra ‘]’ is just being ignored as a sub-group variable name that is never used.

Found another error on line 668 of etc/takeData.tcl:

set m "vme dlts + "${user}@${host}:/$dir"

Check this in tclsh:

% set m "vme dlts + "${user}@${host}:/$dir"
extra characters after close-quote

The if command starting on line 93 of etc/tpmHandler.tcl has an error. Its structure boils down to this, which tclsh doesn’t like either:

% if 1 { puts "yes" } else { puts "no" } end else
wrong # args: extra words after "else" clause in "if" command

The problem is the “end else” at the end.

New problem on line 100 of etc/iop/catas.tcl:

   if {$mode != "frames" && $mode != "gangs" } { error unknown type $mode" }

Missing opening quote for error command.

Another “else if” which should be “elseif” on line 971 of etc/mop/obsolete.tcl:

   } else if {[regexp secondary $targetFile]} {

Error on lines 105-108 of etc/iop/runLog.tcl:

	set Dx [format "%6.2f $Dx]
	set sdx [format "%6.2f $sdx]
	set Dy [format "%6.2f $Dy]
	set sdy [format "%6.2f $sdy]

(missing close quotes)

I think this is the last one: line 134 of etc/camHandler.tcl:

if {$test} { test_ask_timing SHOWREADVOLTS } # 10 minutes

Needs a semicolon before the comment (checked in tclsh … ok).

September 30, 2008

Migrating to SDSS-3 Repository

Filed under: Uncategorized — sdssops @ 10:38 pm

The SDSS-3 code and wiki are hosted at http://trac.sdss3.org/ using the trac software.

SVN Migration

The sdss3 wiki has an svn page. I sent David S my public key last Friday. Try if the following works:

% svn ls svn+ssh://sdss3svn@sdss3.org/repo
The authenticity of host 'sdss3.org (128.3.7.115)' can't be established.
RSA key fingerprint is f2:ab:ce:fa:66:fc:01:5b:1f:bf:66:96:79:9c:39:50.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sdss3.org,128.3.7.115' (RSA) to the list of known hosts.
Password:

I don’t think it should be asking for my pw. Email Michael Jennings (mejennings@lbl.gov) to check that my public key is registered…

What tools are available for migrating a package from one svn repository to another? This seems like a generic problem, except that the directory structures are different at both ends:

  • http://tops.googlecode.com/svn/trunk/
  • svn+ssh://sdss3svn@sdss3.org/repo/tops/trunk/

svnadmin provides a pair of commands that allow the repo history to be transported via a portable ‘dumpfile’:

dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER]] [--incremental]

Dump the contents of filesystem to stdout in a 'dumpfile'
portable format, sending feedback to stderr.  Dump revisions
LOWER rev through UPPER rev.  If no revisions are given, dump all
revision trees.  If only LOWER is given, dump that one revision tree.
If --incremental is passed, then the first revision dumped will be
a diff against the previous revision, instead of the usual fulltext.

Valid options:
  -r [--revision] arg      : specify revision number ARG (or X:Y range)
  --incremental            : dump incrementally
  --deltas                 : use deltas in dump output
  -q [--quiet]             : no progress (only errors) to stderr

load: usage: svnadmin load REPOS_PATH

Read a 'dumpfile'-formatted stream from stdin, committing
new revisions into the repository's filesystem.  If the repository
was previously empty, its UUID will, by default, be changed to the
one specified in the stream.  Progress feedback is sent to stdout.

Valid options:
  -q [--quiet]             : no progress (only errors) to stderr
  --ignore-uuid            : ignore any repos UUID found in the stream
  --force-uuid             : set repos UUID to that found in stream, if any
  --use-pre-commit-hook    : call pre-commit hook before committing revisions
  --use-post-commit-hook   : call post-commit hook after committing revisions
  --parent-dir arg         : load at specified directory in repository

Try a dump from googlecode to see how this works:

% svnadmin dump tops -r HEAD > dump.out
svnadmin: Can't open file 'tops/format': No such file or directory

% svnadmin dump https://tops.googlecode.com/svn/trunk -r HEAD > dump.out
svnadmin: 'https://tops.googlecode.com/svn/trunk' is an URL when it should be a path

It looks like dump needs a unix path to a local repository (format is a top-level file in the repository), but that won’t work with googlecode. I’m still not sure if the dumpfile goes to stdout or some std filename.

There is also a svnsync command that works between two remote repositories, but is not so easy to setup. I think I should aim for a simple one-time migration of the HEAD instead of trying to use tools that can propagate the entire revision history.

Create a local repository for testing at /Users/david/Cosmo/SDSS/Design/repo/svn/. The corresponding URL should then be file:///Users/david/Cosmo/SDSS/Design/repo/svn/. Try to mirror the sdss3 directory structure:

~...repo/test % svn checkout file:///Users/david/Cosmo/SDSS/Design/repo/svn
Checked out revision 0.
~...repo/test % svn mkdir svn/repo
A         svn/repo
~...repo/test % svn mkdir svn/repo/tops
A         svn/repo/tops
~...repo/test % svn ci svn -m 'recreate sdss3 layout'
Adding         svn/repo
Adding         svn/repo/tops

Committed revision 1.
~...repo/test % svn mkdir svn/repo/tops/trunk
A         svn/repo/tops/trunk
~...repo/test % svn mkdir svn/repo/tops/tags
A         svn/repo/tops/tags
~...repo/test % svn mkdir svn/repo/tops/branches
A         svn/repo/tops/branches
~...repo/test % svn ci svn -m 'add standard project layout'
Adding         svn/repo/tops/branches
Adding         svn/repo/tops/tags
Adding         svn/repo/tops/trunk

Committed revision 2.

Now try importing to this test repo from my checked-out googlecode:

~...Design/dev % svn import tops file:///Users/david/Cosmo/SDSS/Design/repo/svn/repo/tops/trunk -m 'initial import'
Adding         tops/sdss3
Adding         tops/sdss3/config.ini
Skipped 'tops/sdss3/.svn'
Adding         tops/sdss3/__init__.py
Adding         tops/sdss3/design
Adding         tops/sdss3/design/general_cases.py
Adding         tops/sdss3/design/tcc_states.py
Adding         tops/sdss3/design/path.py
Adding         tops/sdss3/design/marvels_states.py
Skipped 'tops/sdss3/design/.svn'
Adding         tops/sdss3/design/__init__.py
Adding         tops/sdss3/design/model.py
Adding         tops/sdss3/design/web
Adding         tops/sdss3/design/web/marvels_states.html
Adding         tops/sdss3/design/web/tcc_states.html
Adding         tops/sdss3/design/web/marvels_states.dot
Adding         tops/sdss3/design/web/tcc_states.dot
Skipped 'tops/sdss3/design/web/.svn'
Adding         tops/sdss3/design/web/cases.css
Adding         tops/sdss3/design/web/sdss3model.dot
Adding         tops/sdss3/design/web/glossary.html
Adding         tops/sdss3/design/web/statechart.css
Adding         tops/sdss3/design/web/glossary.css
Adding         tops/sdss3/design/web/statechart.js
Adding         tops/sdss3/design/marvels_cases.py
Adding         tops/sdss3/tcc
Skipped 'tops/sdss3/tcc/.svn'
Adding         tops/sdss3/tcc/__init__.py
Adding         tops/sdss3/tcc/broadcast.py
Adding         tops/sdss3/tcc/session.py
Adding         tops/sdss3/tcc/message.py
Adding         tops/sdss3/tcc/listener.py
Adding  (bin)  tops/.sconsign.dblite
Skipped 'tops/.svn'
Adding         tops/start.py
Adding         tops/__init__.py
Adding         tops/SConstruct
Adding         tops/core
Adding         tops/core/protobuf
Skipped 'tops/core/protobuf/.svn'
Adding         tops/core/protobuf/logging.proto
Adding         tops/core/protobuf/archiving.proto
Adding         tops/core/config.ini
Adding         tops/core/utility
Adding         tops/core/utility/options.py
Skipped 'tops/core/utility/.svn'
Adding         tops/core/utility/__init__.py
Adding         tops/core/utility/name_graph.py
Adding         tops/core/utility/state_chart.py
Adding         tops/core/utility/html.py
Adding         tops/core/utility/config.py
Adding         tops/core/utility/astro_time.py
Adding         tops/core/utility/data.py
Adding         tops/core/utility/secret.py
Skipped 'tops/core/.svn'
Adding         tops/core/__init__.py
Adding         tops/core/design
Adding         tops/core/design/use_cases.py
Skipped 'tops/core/design/.svn'
Adding         tops/core/design/__init__.py
Adding         tops/core/design/domain_model.py
Adding         tops/core/network
Adding         tops/core/network/telnet_test.py
Adding         tops/core/network/telnet.py
Skipped 'tops/core/network/.svn'
Adding         tops/core/network/logging
Adding         tops/core/network/logging/logging_pb2.py
Skipped 'tops/core/network/logging/.svn'
Adding         tops/core/network/logging/record.py
Adding         tops/core/network/logging/__init__.py
Adding         tops/core/network/logging/test.py
Adding         tops/core/network/logging/producer.py
Adding         tops/core/network/logging/server.py
Adding         tops/core/network/logging/web
Adding         tops/core/network/logging/web/logwatch.css
Adding         tops/core/network/logging/web/logwatch.js
Skipped 'tops/core/network/logging/web/.svn'
Adding         tops/core/network/logging/web/logwatch.html
Adding         tops/core/network/__init__.py
Adding         tops/core/network/client_server_test.py
Adding         tops/core/network/webserver.py
Adding         tops/core/network/server.py
Adding         tops/core/network/naming.py
Adding         tops/core/network/command.py
Adding         tops/core/network/client.py
Adding         tops/core/network/proxy.py
Adding         tops/core/network/archiving
Skipped 'tops/core/network/archiving/.svn'
Adding         tops/core/network/archiving/archiving_pb2.py
Adding         tops/core/network/archiving/record.py
Adding         tops/core/network/archiving/__init__.py
Adding         tops/core/network/archiving/producer.py
Adding         tops/core/network/archiving/server.py
Adding         tops/core/network/archiving/web
Skipped 'tops/core/network/archiving/web/.svn'
Adding         tops/core/network/archiving/web/archiver.html
Adding         tops/core/network/archiving/web/archiver.css
Adding         tops/core/network/archiving/web/archiver.js
Adding         tops/core/network/web
Adding         tops/core/network/web/library.js
Skipped 'tops/core/network/web/.svn'
Adding         tops/core/network/web/jquery-ui-personalized-1.5.2.min.js
Adding         tops/core/network/web/jquery-ui.js
Adding         tops/core/network/web/flora.dialog.css
Adding         tops/core/network/web/jquery.js
Adding         tops/core/network/web/styles.css
Adding         tops/core/network/web/jquery-1.2.6.min.js
Adding         tops/run_tests.py
Adding         tops/start

Committed revision 3.

Import includes a commit so needs a log message. My ‘.pyc’ files were automatically ignored, presumably because of my source svn config. The import command seems to know that it should ignore .svn files. The only file that shouldn’t be there is:

tops/.sconsign.dblite

Test that I can check this all out again:

~...repo/test % svn update svn
A    svn/repo/tops/trunk/sdss3
A    svn/repo/tops/trunk/sdss3/config.ini
...
A    svn/repo/tops/trunk/core/network/web/jquery-1.2.6.min.js
A    svn/repo/tops/trunk/run_tests.py
A    svn/repo/tops/trunk/start
Updated to revision 3.
~...repo/test % ls svn/repo/tops/trunk/
svn/repo/tops/trunk:
SConstruct     __init__.py    core/          run_tests.py   sdss3/         start*         start.py

Try tagging this:

~...repo/test % svn copy file:///Users/david/Cosmo/SDSS/Design/repo/svn/repo/tops/trunk file:///Users/david/Cosmo/SDSS/Design/repo/svn/repo/tops/tags/V0.4 -m 'tag release'

Committed revision 4.

Looks ok!

Friday 3 Oct 2008 Update

I never heard back from Michael Jennings but svn access seems to be working now:

% svn ls svn+ssh://sdss3svn@sdss3.org/repo
boss/
marvels/
photolog/
photoop/
platedesign/
platelist/
rc3/
sas/
sdssfake/
segue/
template/

Checkout the top-level repo/ directory so I can add my top-level dirs:

% pwd
/Users/david/Cosmo/SDSS/Design/trac
% svn co --non-recursive svn+ssh://sdss3svn@sdss3.org/repo/ repo
Checked out revision 6035.
% svn mkdir repo/tops
A         repo/tops
% svn mkdir repo/tops/trunk
A         repo/tops/trunk
% svn mkdir repo/tops/tags
A         repo/tops/tags
% svn mkdir repo/tops/branches
A         repo/tops/branches
% svn ci repo -m 'create empty dirs for new package'
Adding         repo/tops
Adding         repo/tops/branches
Adding         repo/tops/tags
Adding         repo/tops/trunk

Committed revision 6036.

Wait until I have a revision suitable for a V0.5 tag and then use that to synchronize between googlecode and sdss3. Note that sdss3 policy states that:

Tag names should be of the form “vN_M_P” (with “_P” optional). “N” should change only with major changes in the code.

so I will tag v0_5 as the twin of googlecode’s V0.5.

Before I check in the code, I also need to figure out the best way to ignore the sconstruct db file. Read the redbook discussion of ignoring files…there are two methods:

  1. the global-ignores in the run-time config: this only applies to a particular svn user or perhaps all users on one host: this is what I did for the .pyc files
  2. set the svn:ignore property with ’svn propedit svn:ignore …’ which only applies to a single directory (but patterns like *.pyc are ok). propedit opens up an editor so i need to configure that…use the –editor-cmd option or, better, modify my ~/.subversion/config to use /usr/bin/mate (TextMate) … done (I also added *.dblite to global-ignores)

In the end, I will use the local config file for *.pyc and *.dblite and then propedit for specific files (like the auto-generated html ones) that should be ignored.

Should I update all the headers to change this line?

# This project is hosted at http://tops.googlecode.com/

perhaps into this?

# This project is hosted at sdss3.org and tops.googlecode.com

Mon Oct 6 Update

Did the initial import to sdss3.org:

% svn import tops svn+ssh://sdss3svn@sdss3.org/repo/tops/trunk/ -m 'initial import of v0_5 candidate'
Adding         tops/sdss3
Adding         tops/sdss3/config.ini
Skipped 'tops/sdss3/.svn'
Adding         tops/sdss3/__init__.py
...

This also grabbed my auxiliary files (protobuf, etc). Next, check out tops/trunk so I can remove these:

~...Design/trac % svn rm tops/sdss3/design/web/marvels_states.html
D         tops/sdss3/design/web/marvels_states.html
~...Design/trac % svn rm tops/sdss3/design/web/tcc_states.html
D         tops/sdss3/design/web/tcc_states.html
~...Design/trac % svn rm tops/sdss3/design/web/marvels_states.dot
D         tops/sdss3/design/web/marvels_states.dot
~...Design/trac % svn rm tops/sdss3/design/web/tcc_states.dot
D         tops/sdss3/design/web/tcc_states.dot
~...Design/trac % svn rm tops/sdss3/design/web/sdss3model.dot
D         tops/sdss3/design/web/sdss3model.dot
~...Design/trac % svn rm tops/sdss3/design/web/glossary.html
D         tops/sdss3/design/web/glossary.html
~...Design/trac % svn rm tops/core/network/logging/logging_pb2.py
D         tops/core/network/logging/logging_pb2.py
~...Design/trac % svn rm tops/core/network/archiving/archiving_pb2.py
D         tops/core/network/archiving/archiving_pb2.py

Try setting svn:ignore property … my editor setting didn’t work, change it to ‘editor-cmd = /usr/bin/mate –wait’ … ok. Just ignore the protobuf *_pb2.py files for now.

Tag on each repository:

~...Design/trac % svn copy svn+ssh://sdss3svn@sdss3.org/repo/tops/trunk svn+ssh://sdss3svn@sdss3.org/repo/tops/tags/v0_5 -m 'matches V0.5 on tops.googlecode.com'

Committed revision 6042.

~...Design/dev % svn copy https://tops.googlecode.com/svn/trunk https://tops.googlecode.com/svn/tags/V0.5 -m 'matches v0_5 on sdss3.org'

Committed revision 303.

Wiki Migration

Are the trac wiki pages in svn and can I import/commit?

Are the googlecode pages compatible with the format expected by trac? Presumably the wiki markup conventions are similar, but perhaps the files in svn are not just straight ascii?

Blanton has a wiki page discussing this and providing python programs postWiki and getWiki … uggh, these are faking an interactive session with the wiki editor! I guess the wiki uses its own database and not svn :-(

September 19, 2008

TODO List

Filed under: proxy — sdssops @ 8:47 pm

Short-term priorities:

  1. Use file-based config parameters … done
  2. Implement batch process launcher … done
  3. Figure out how to share UDP port on sdsshost2 … not possible?
  4. Proxify tcc.session and connect to archive records
  5. Skeleton marvels proxy
  6. Command-line archive “get”
  7. Implement automatic “CurrentState” channel in proxy
  8. First pass at proxy command interface

Longer-term projects:

  • Better define validity of archive channels based on their proxy’s state
  • Archive channel processors (deadband, running avg, …)
  • Database backend
  • Hierarchical display of channels in archive web client
  • Process watcher
  • Partition manager (combined with process watcher?)

Run-time Configuration

A project needs to be able to override the configuration used by core services like the archiver and logger. This means that the core software needs to call config.initialize(project_path). Where does it get project path from?

Could read it from $CWD/tops-config.ini or pass it in as a command-line parameter to the launch script.

In the first case, there is a bootstrap problem since config.initialize() must be called in order to read tops-config.ini but then it is too late to set the project path!

The second solution requires that the launch command parameter is propagated to all processes that it runs, e.g., via a standard args parsing module?

I have the run-time config mostly working now. The remaining items are to:

  • have the logging producer read the config parameters … done
  • fix up logging.test to read config parameters … done
  • implement variable substitution in network.webserver and use it to apply web_title

Process Launcher

Lives in tops/start.py and takes an optional project name parameter. Put the code to parse standard command-line options into core/utility/options.

How to stop all processes in reverse order? For now, just store a reversed list of pids in a file whose path is a run-time config option and print out the necessary ‘kill’ command.

Still to do: cleanup the listener/proxy startup output:

start: service tops.sdss3.tcc.listener is pid 19189
tops.core.network.logging.producer: using source name "tcc.listener"
INFO:root:Starting proxy
tops.core.network.logging.producer: trying to connect to the logging server...
Success!
INFO:root:Starting archive client
INFO:root:Will listen for UDP broadcasts on port 1200
bindUDPSocket: binding a socket using the SO_REUSEPORT option
setting socket options in startProtocol()
DEBUG:root:Started in state "WAITING"

The “INFO:root:…” and “DEBUG:root:…” messages are actually duplicates of messages being sent to the logger. Why does this happen and why doesn’t it happen for the logger or archiver?

The current proxy startup logic is:

  1. proxy.initialize()
    1. config.initialize()
    2. archiver.initialize()
  2. declare proxy (defines states, archive records)
  3. proxy.start()
    1. infer name from top-level state, e.g. TCC_LISTENER -> tcc.listener
    2. logger.start(name)
      1. reconfigures built-in logger classes but does not send header
    3. archiver.start(name)
      1. sends header
    4. set initial state
    5. start twisted reactor main loop

Note that the logger does not send its header until the first message is actually sent, in its createSocket method. The reason for this is to allow the producer client to automatically reconnect in case of a dropped connection. It would be cleaner if the logger and archiver producers had the same startup logic, but that’s a bigger project.

For now, provide the service name to the logger and archiver initialize() methods so that logging works right from the start. This decouples the top-level state name from the service name. Should the service name be obtained from an INI file? This would require passing the relevant INI section name to the initialize() methods. Perhaps these should be required to be the same (section name = service name) so that initialize can look up any generic service parameters in config that it needs…ok.

The new startup logic that I am aiming for is then:

  1. proxy.initialize(name) [ name = service name = config section name ]
    1. remember name
    2. config.initialize()
    3. logger.initialize(name)
    4. archiver.initialize(name)
  2. declare proxy
  3. proxy.start()
    1. archiver.start()
    2. set initial state
    3. start twisted reactor main loop

Plan of attack:

  • rename logger start(name) to initialize(name) and update network.logging.test … done
  • move archiver initialization code from start(name) to initialize(name) … done
  • proxy remembers its name in proxy.initialize() … done
  • proxy.start() no longer infers name, does not call logger.start() and calls archiver.start() methods without a name parameter … done
  • update listener() to call initialize() with a name and don’t expect a return value … done

Testing … ok … committed revision 154

UDP Port Sharing

Refer to the setsockopt man page for the syntax that socket.setsockopt(…) needs:

  • level is SOL_SOCKET
  • option is SO_REUSEADDR and/or SO_REUSEPORT
  • any value needed?

From the man page:

SO_REUSEADDR indicates that the rules used in validating addresses supplied in a bind(2) call should allow reuse of local addresses.  SO_REUSEPORT allows completely duplicate bindings by multiple processes if they all set SO_REUSEPORT before binding the port.  This option permits multiple instances of a program to each receive UDP/IP multicast or broadcast datagrams destined for the bound port.

Try the PORT version first. This seems to work with Fritz’s tcc_packet.py:

s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEPORT,1)

How do I set socket options in twisted? Right now, all I do is call the reactor’s listenUDP method with a port number and my DatagramProtocol subclass. I’ve seen examples where setsockopt is called in DatagramProtocol.startProtocol() but mine is never called. What is happening behind the scenes?

  1. reactor.listenUDP(port,protocol)
    1. p = twisted.internet.udp.Port(port, protocol,…)
    2. p.startListening()
      1. self._bindSocket()
        1. skt = self.createInternetSocket() [implemented in twisted.internet.base.BasePort]
        2. skt.bind(…)
      2. self._connectToProtocol()
        1. protocol.makeConnection()
        2. self.startReading()

The REUSEPORT option needs to be set after the socket has been created but before it is bound, so it looks like I need to customize the implementation of twisted.internet.udp.Port._bindSocket(). This is ugly since I am duplicating twisted code in a semi-private method, but I don’t see any better solution. Allowing socket options to be customized before binding would be a good contribution for me to make to a future twisted release…perhaps via a protocol configureSocket() method that the port calls in _bindSocket via its self.protocol instance?

Sent an email to Fritz asking him to make this change in his tcc_packet and restart on sdsshost2. Fritz replies that REUSEPORT is not implemented on Solaris! I tested whether REUSEADDR would work just as well but, as expected, it does not allow a second process to bind to the same port (although this would apparently be ok for multicast).

Fritz says that I can use sdssfiles1 instead of sdsshost2. Here is the uname -a output from each host:

Linux sdssfiles1.apo.nmsu.edu 2.6.9-42.0.3.ELsmp #1 SMP Thu Oct 5 16:29:37 CDT 2006 x86_64 x86_64 x86_64 GNU/Linux
Linux sdsshost2.apo.nmsu.edu 2.4.21-32.0.1.ELsmp #1 SMP Wed May 25 16:24:34 CDT 2005 i686 athlon i386 GNU/Linux

Hmm, I assumed that sdsshost2 was solaris because it didn’t have the SO_REUSEPORT socket option, but I guess linux doesn’t either!? It looks like sdssfiles1 is running a more recent kernel and is 64-bit compared with sdsshost2’s 32-bit i686.

Fritz has installed all of the tools I need on sdssfiles1 (!) but I cannot see its ports from uci.edu. Ask Fritz if its possible to allow TCP traffic on 8080 and 8081.

As an alternative for this week’s demonstration, can I see the 3.5m TCC broadcasts on sdsshost2? The port number is 1235. I found tcpdump on sdsshost2 in /usr/sbin/ but I don’t have sudo privs.

Proxify tcc.ssession

  • I already have a tcc.session section in config.ini but with the service disabled. Enable it now … first snag is that the password cannot be read from stdin when the process is started in the background
  • Move __main__ code into config() … done
  • Declare a simple state machine with a single “vms” record in __main__ … done
  • Trap login failed condition via a proxy state transition
  • Convert print statements to log.msg(), etc
  • Enable periodic updaters
  • Trap command buffer overflow via a proxy state transition
  • Define more monitor records

Possible solutions for getting the tcc password:

  1. connect the tcc.session stdin to start’s stdin and increase the delay before start moves to the next service (or exits if tcc.session is the last service to launch)
  2. put the password in plain text in a config file that must be specified via –config (so the password is not in svn)
  3. put an encrypted version of the password in the tops.sdss3 config file and add a config.getsecret() method that decrypts strings using a master password that config.initialize() prompts for and passes to subprocess via the command line

The first solution is kludgy because it imposes an artificial time limit for entering the password and does not easily support sensible error handling in case of a timeout

The third solution is elegant but compromises the password by passing the master password in plaintext via the command line to all subprocesses.

The second solution is easy and perhaps the best for now. The config file should be read-only by owner: chmod 0600 …

How do TUI/TRON and xOP handle this?

Perhaps there is a good 4th solution: the config file can identify any secret data that a service requires so that start could collect it … no, this still has the problem of how to transmit secrets to subprocesses. If I can solve this, then #3 is better.

Try sending the master password via a stdin pipe to each subprocess. options.initialize() can read this and make it available to config.

Python does not have any built-in encryption library so I either need to install a 3rd-party one (also at APO) or else use a standard command-line tool.

I have installed PyCrypto 2.0.1 on my laptop and at APO. The docs recommend AES encryption for general-purpose use. AES needs a key that is 16, 24, or 32 bytes long and is not restricted to ascii characters. The data must be padded to a multiple of 16 bytes.

Create a new tops.core.utility.secret to interface with PyCrypto and support start, config … ok

I just tested tcc.session on sdssfiles1 … it runs fine (!), producing lots of output (still need to clean this up) … it eventually died with:

TCCException: TCC: bad keyword ' no more errors reported for this user until output resumes"'

Looking back at my earlier TCC tests, the actual output line probably looked something like this:

0 0 W UserNum=1; MsgsLost=1; Text="user mailbox full; no more errors reported for this user until output resumes"

The problem is that I am splitting the line into tokens with line.split(‘;’) but this is fooled by the semicolon embedded within the string.

Older Posts »

Blog at WordPress.com.