Dear Friends,
Hi!
I am Choudhary at this end, I am looking forth for data which would
reveal or form a sort of collective map of chemical compositions 'as
detailed and as abundance as possible' if say we were to build one for
the three d astro-space in coordinate space that surrounds us.. based
upon all the observations undertaken... (lets keep time out of the
picture for the moment).  The hubble telescope has collected data over
a decade I need the raw data collected by it. This question has
emanated today when I had been through a question posed previously on
google by someone who was inquiring about presence of water in deep
space... and well he had received a comment which directed him to some
european site... I am preferring experts help to save some time...

I am NOT an expert, but to the extent that you research this yourself,
you might try searching on the phrase "interstellar medium."

Choudhary,

Astonishingly, all of the data collected by the Hubble Space Telescope
is available to the public for free in the Hubble Data Archive(HDA).
Here is the link:

http://archive.stsci.edu/hst/getting_started.html

Apparently, as of September 1, 2002, this represents over 9.5
Terabytes of data. I suspect that even the most wildly enthusiastic
Google researcher is going to be unable to aggregate that for you for
four dollars. The good news is the HDA makes it easy (well, easier)
for you to retrieve the data by providing a simple web interface.

Oddly enough, I think it would be possible to do what you suggest -
make a three dimensional map of the composition of interstellar space
centred on the Earth. You would have to focus on the spectrograph
data, and you would have to decide which parts of it represented
stellar features, and which parts represented interstellar absorption
and, perhaps, emission. Measuring distance would be a panic, but you
could at least compare the results from near stars to those from far
stars to get a rough idea.

I think that there may be other platforms more suited to the task,
like WHAM and CHIPS (mentioned here:
http://www.olemiss.edu/courses/astr104/Topics/Interstellar-L.html) but
you could probably make a stab at it with Hubble.

This would be a pretty interesting PhD thesis project, or maybe a post-doc.

Your question has made me realize what an amazing world we live in:
sitting in my living room I have access to the entire data output of a
space telescope. Who needs television?

Alan Kali

Thanks Alan,
       For the link, All I can say is completing a Graph in coordinate
system would be a challenge and has to be undertaken... to pinpoint a
model of our cosmos..... or say to grow a model for our cosmos....
However the problem that is going to surface is handling of the data
.. could you kindly suggest a cheapest form of, memory storage
devices... say which would contain handle in the entire 9.5 terabytes
of data... or some way in which I could best process the data
available and have my graph ready to the detail observed by hubble
lets say.... Any suggestions on how could the entire data be converted
in the cheapest way as possible... I am asking it to be cheapest since
well I would be placing money from my personal pocket... Well by the
way based upon your last statement kindly permit me to add on an
extension as what about the mind.... ?

What did you mean by the statement below... I do not know about WHAM or CHIPS...
I think that there may be other platforms more suited to the task,
like WHAM and CHIPS (mentioned here:

Choudhary,

Terabyte-level storage on a budget is a big problem. I know of no way
to store that much data conveniently with the resources available to
most individuals. You may be able to find a university near you that
mirrors the data, and can allow you some access to it. Alternately,
you might wish to post the question of, "How to store 9.5 terabytes of
data economically?" as another Google Answers question. You may wish
to specify that the data must be on-line, rather than archival, since
you will need ready access to it for your research.

Happily, your data storage requirements should be drastically reduced
by the fact that you will not require all 9.5 terabytes for the
project you describe. Hubble is composed of many instruments, all (at
some point - many are quiet now) producing data. Much of that data
will have no direct relevance to your project. This will include
raster scans from imaging devices: very bulky stuff. I have no idea
how much you shall ultimately need to process, but I think it is safe
to say that it will be a small fraction of the total take.

Delightfully, the HDA site seems to be up most of the time, and I know
of no plans to make it unavailable in the future. That means you
should be able to go over to HDA and start extracting small amounts of
test data and importing them to your graphing application. In that
case, you need never store more data than you are working on at a
given time, plus the output set. This is both economical and
practical. You can use HDA for your data storage and only extract what
you need!

The great challenge will be not storing, but filtering the data in a
way that is both accurate and useful. The thinking you have to do
there is along the lines of: What does this particular instrument give
me? How do I extract what I need? How do I ensure that my data is
sane? and How do I map it? This looks like a lot of work to me. I
would begin by reading up on the scientific and technical capabilities
of each Hubble instrument, assuming you decide to use the Hubble data.
The first link I gave you before should get you started.

By the way, should you reach the stage of actually wanting to map some
Hubble data, I can recommend a great program: GMT. It is free and
source is available! This is probably the best known program for the
kind of thing you are trying to do, so there is plenty of help
available both locally and on the net. I am using it for one of my
projects right now.

GMT is at: http://gmt.soest.hawaii.edu/

In answer to your other question:
WHAM = Wisconsin Hydrogen-Alpha Mapper
CHIPS = Cosmic Hot Interstellar Plasma Spectrometer

These projects deal with the warm ionized or plasma component of
interstellar matter. They each have a link from the web site I
mentioned. I don't happen to know about the availability of raw data
from either of them, but you might find the maps (especially at the
WHAM site) interesting, since these represent some component of what
you wish to accomplish.

Don't forget to check out the Cometary and Interstellar Dust Analyzer
(CIDA) currently on Stardust, either, since that is the only
instrument (so far as I know) that has given us a direct measurement
of interstellar components. It's way too early for any data return
from Stardust, but it is a project you might wish to be aware of. The
link is a little further down the same page as WHAM and CHIPS.

In case you are missing part of my previous post, here are the links again:
HDA is at - http://archive.stsci.edu/hst/getting_started.html
Good link site is at -
http://www.olemiss.edu/courses/astr104/Topics/Interstellar-L.html
WHAM is at - http://www.astro.wisc.edu/wham/
CHIPS is at - http://chips.ssl.berkeley.edu/

Best Regards,

Alan Kali

Thanks Alan.... for the detailed reply... and your helpful advice.....
However, since you are already working with GMT and adapted to it...
Could it be possible for you to kindly accept my request and process a
few of the data of HDA... However for entire 9.5 Terabytes of data
definitely price for processing is the problem... well..  I would be
glad for you to kindly quote your price for entire and or part of the
data.... I am pretty much sure that you are at the moment handluing
GMT... so it would be easier for you... the format in which I am
prefering to have the data is in terms of numbers (space time
coordinates) which do even get converted to a simple mpeg... mpeg
would be for mere observations of the evolutionary process and well
the format in terms of numbers data would be used for comparision of
the said results with the predicted ones..... I bid you bye for now...
since have to post  the  question for handling the data......

Choudhary,

I am not sure that your project is at the implementation stage at this
point. Before extracting data, one would need to understand the goals
of the project in detail, the nature of the data collected by each of
the Hubble instruments as it relates to your project, whether or not
this data would be suitable to your goals, whether other intstruments
have produced more relevant or appropriate data, whether that data is
available at this point, and whether someone else has achieved
something substantially similar already.

This is not something that one could undertake lightly. Even for
someone active in the field of mapping interstellar constituents
(which I am not - my GMT project involves surface topography) a
project like this would take many months of consideration and research
before reaching the implementation stage.

I don't think there is any shortcut here. Science is often painful, or
at least tedious. Unless you have the resources of a funding agency,
your only option will be to do this yourself.

Best Regards,

Alan Kali

P.S.

Choudhary,

Just in case you DO have the resources of a funding agency, I have a
couple of hungry grad students kicking around.

They'd kick me if it turns out you're actually from NSERC or something.

Alan Kali

Dear Alan,

Hi!

At this very moment I have nothing more than $ 500, well all i can say
is that I would prefer to store the data by having them written in
CD's say one cd costs Rupees 8.00 and has 670 MB so a thousand CD's
would be having 0.67 TB and well the cost for that would be $ 200, The
nature of data that I am looking for has been described below......

Say if we can work out the evolution stages of the Universe then for
us to have it for comparision with some kind of data I would be
looking for the evolutionary kind of data (no matter for whatever
small time frame) that has been collected by hubble or any other
projects .... This data I would need to have it in terms of spatial
coordinate system and the time as a function... alongwith the measured
data ... thats it nothing more than this for the moment keeping in
mind the low budget work out... of data only to the tune of 0.67 TB...
However could you reconfirm that the space required would be 9.5 TB
since, that much of space would be raw data what about the processed
data .... ? Finally your comments on what would be the total
expenditures towards this work... Say we need the complete
evolutionary nature as detailed as possible...

P.S. 
You can be assured that I am not from NSERC, I am from India ....

Choudhary,

I am impressed by, and respectful of, your enthusiasm for science, but
I must advise you to be cautious. Despite the approach taken by many
funding agencies and perhaps mistakenly assumed by the general public,
scientific problems rarely respond to the process of "throwing money
at them".

I would be lying if I said that I understood your approach to the
problem from so little information as can be presented through this
medium, and I doubt that anyone could say otherwise. Anyone who agreed
to take money from you to work on this project would come under
suspicion of having questionable motives.

I would not like to see you get hurt, and I would prefer that you keep
your delightful enthusiasm for science. Therefore I cannot recommend
strongly enough that you put your money toward an academic program
that will take you into a field where you can begin to answer these
questions. I strongly suggest that you begin by finding a local
academic advisor whom you trust, and who may guide you in terms of
academic choices.

I responded to your original comment because I felt that I could point
you in the direction of scientific source material and tools that are
both free and freely available. If your computer and Internet
connection are capable of supporting it, you may do this work
gradually without additional expense and educate yourself as to the
methods of science as you go.

And yes, I can confirm that the HDA web site describes the accumulated
raw data from Hubble as being 9.5 terabytes as of September 1, 2002.
All data goes through a proprietary period of one year before being
routinely available to the public, so this probably represents the
released data at the last update, rather than the entire amount. Once
again, I cannot imagine a way that you could process all of this data
and derive only a single useful map. I am sure that you need only a
fraction of it for any project involving the distribution of
interstellar components.

Best Regards,

Alan Kali

Thanks Alan, 

For the Seniorly advices, the question that remains unanswered..... is
to how much $ would it take for getting a sizable portion of the data
processed..... I would prefer having this knowhow say incase I obtain
funding.... then we could communicate further on this topic........
thanks for cautioning that funds do not get answers in science
.......... However all I am looking for is to have something in hand
ready for comparision and with my minimal investment of time.... or
effort .... rather then get it readied at later stage.... However, I
have a question... definitely it would appear Naive momentarily... but
well let it appear and since I find you a bit more than interesting I
pose it to you........


a) Is it that a modeller who models the cosmos would need the
experimental data collected by Hubble or any other devices to prove
his model ?


I hope we meet some day... till then ....Cheers !!!!!!!!... Enjoy ....

Kindly confirm once you have read this message...

Choudary,

I can process all of the data from the Hubble for you by setting up a
Perl script to query the HDA database for the binary string:

010000010110000101100001011100100110011101101000001000010010000100100001

This will cost nothing except time, cycles and bandwidth, but it will
not be very useful in any context other than proving that the most
important and difficult thing is not processing the data, but HOW you
process the data.

As to HOW to process the data, the best answer (sorry, "comment") I
can make is by referring you to one of my favourite quotations from
popular culture:

This is from "Star Trek: The Next Generation", the episode called,
"The Ensigns of Command". Captain Picard has been badgering Geordi
Laforge, the chief engineer, to resolve a very difficult problem with
the transporters, and the captain will not take "no" for an answer.

GEORDI: Captain, we can do it! We can modify the transporters.
PICARD: Excellent.
GEORDI: It'll take fifteen years, and a research team of a hundred --
PICARD: Mister La Forge, I believe we will postpone.

As to whether a modeler would require data from the Hubble to validate
a model, I suppose that depends upon what is being modeled.

Best Regards,

Alan Kali

Dear Alan,

Hi!

As far as the question goes, What is being modelled... Well its the
cosmos and as fine tuned as it is, I would prefer to say, rather than
as could be thought... Anyway... still the question that remains
unanswered is how much would it cost to process the data... say we
place in a two years time frame for processing the data then what
would the cost be ( or how quick could we process this data and the
cost involved for it) ? Well kindly give me a statistics of the cost
as per time .. Where time is a resonable one....... Say by 2007 or
2008 I would be testing the predictions of my model to the
experimental data that I am requesting you to process collectively or
individually. Its an uphill task that has been undertaken and started
with..... So, as a form of conclusion... could we get down to
business....

Dear Alan,
Hi!
I have not got your reply ... Could you kindly confirm having gone
through this note ...

Hi,

Just a note to say there are dozens of scientists (and Ph.D. students)
working on this already! Here are some example results:

http://www.berkeley.edu/news/media/releases/2003/05/29_space.shtml
http://ukads.nottingham.ac.uk/cgi-bin/nph-bib_query?bibcode=1998ApJ...500..525S&db_key=AST&high=40bd60ade402607
http://www.star.le.ac.uk/wd/ism.html
http://www.daviddarling.info/encyclopedia/L/Local_Bubble.html
http://simbad.u-strasbg.fr/cgi-bin/aasbib?2000ApJ...534..825R

Thanks 

RNT20-ga

From Choudhary!!!!!

Thanks 

RNT the work suggested has eased off a certain chunk of load .... 

Choudhary!!!!!