What is the secret protocol used by realplayer and the now illegal
StreamBoxVCR to download real media files? (Only those with the copy
protection bit cleared of course)

Hello arjen, 
 
RealNetworks Inc.’s Realaudio technology premiered in 1995 delivering
streaming audio over the World Wide Web.  This technology has improved
over the last few years incorporating high-quality protocols such as
Sony’s ATRAC3 to deliver streaming audio and video, and compatibility
with synchronous real-time data delivery (i.e. RTP):
&lt;a href=&quot;<a href="http://www.sony.co.jp/en/Products/ATRAC3/wat3/gijyutu.html">http://www.sony.co.jp/en/Products/ATRAC3/wat3/gijyutu.html</a>&quot;&gt;<a href="http://www.sony.co.jp/en/Products/ATRAC3/wat3/gijyutu.html">http://www.sony.co.jp/en/Products/ATRAC3/wat3/gijyutu.html</a>&lt;/a&gt; 
&lt;a href=&quot;<a href="http://www.realnetworks.com/solutions/leadership/realaudio.html">http://www.realnetworks.com/solutions/leadership/realaudio.html</a>&quot;&gt;<a href="http://www.realnetworks.com/solutions/leadership/realaudio.html">http://www.realnetworks.com/solutions/leadership/realaudio.html</a>&lt;/a&gt; 
&lt;a href=&quot;<a href="http://www.cs.columbia.edu/~hgs/rtp/">http://www.cs.columbia.edu/~hgs/rtp/</a>&quot;&gt;<a href="http://www.cs.columbia.edu/~hgs/rtp/">http://www.cs.columbia.edu/~hgs/rtp/</a>&lt;/a&gt; 
&lt;a href=&quot;<a href="http://www.rtsp.org/">http://www.rtsp.org/</a>&quot;&gt;<a href="http://www.rtsp.org/">http://www.rtsp.org/</a>&lt;/a&gt; 
 
The protocol used to deliver Realmedia files over the web nowadays is
not really a “secret” protocol.  In fact, RealNetworks publishes white
papers of most technology it sells and numerous websites offer
technical data:
&lt;a href=&quot;<a href="http://www.realnetworks.com/solutions/leadership/whitepapers.html">http://www.realnetworks.com/solutions/leadership/whitepapers.html</a>&quot;&gt;<a href="http://www.realnetworks.com/solutions/leadership/whitepapers.html">http://www.realnetworks.com/solutions/leadership/whitepapers.html</a>&lt;/a&gt; 
&lt;a href=&quot;<a href="http://www.cs.columbia.edu/~hgs/rtsp/">http://www.cs.columbia.edu/~hgs/rtsp/</a>&quot;&gt;<a href="http://www.cs.columbia.edu/~hgs/rtsp/">http://www.cs.columbia.edu/~hgs/rtsp/</a>&lt;/a&gt; 
 
One such white paper discusses the Real-Time Streaming Protocol (RTSP)
used in conjunction with the Real System Server.  This is most likely
the protocol you use when previewing Realplayer music clips, or video
clips on news sites.
&lt;a href=&quot;<a href="http://docs.real.com/docs/rn/RealSystem_iQ_RTSP.pdf">http://docs.real.com/docs/rn/RealSystem_iQ_RTSP.pdf</a>&quot;&gt;<a href="http://docs.real.com/docs/rn/RealSystem_iQ_RTSP.pdf">http://docs.real.com/docs/rn/RealSystem_iQ_RTSP.pdf</a>&lt;/a&gt; 
 
Since there are two ways of streaming Realmedia files (through RTSP
and HTTP) I will deal specifically with RTSP since your question
concerns saving local copies of streamed media.  Think of the RTSP
spec as being the initiator and controller of the stream, and the Real
Time Transport Protocol (RTP) mentioned above as the actual deliverer.
 Realnetworks uses their version of a RTSP spec to control how the
media file gets received on your computer.  The Realplayer program on
your computer communicates with the server and agrees on the RTSP
spec.  By not incorporating “saving” as a feature in your Realplayer
program, the stream is decoded, processed, and presented, but not
stored.  This why cached copies do not work, since they will either
(a.) contain no data or (b.) attempt to connect back to the server to
confirm the RTSP spec.
 
The Streambox VCR program you refer to was a product (along with
Streambox Ferret)  developed by Seattle’s Streambox.com to allow users
to saved local copies of streamed media files.  It lost a lawsuit
filed by Realnetworks, Inc. in 2000 and has not been allowed to
distribute the StreamboxVCR program.
&lt;a href=&quot;<a href="http://zdnet.com.com/2100-11-517481.html?legacy=zdnn">http://zdnet.com.com/2100-11-517481.html?legacy=zdnn</a>&quot;&gt;<a href="http://zdnet.com.com/2100-11-517481.html?legacy=zdnn">http://zdnet.com.com/2100-11-517481.html?legacy=zdnn</a>&lt;/a&gt; 
&lt;a href=&quot;<a href="http://news.com.com/2100-1023-245482.html?legacy=cnet">http://news.com.com/2100-1023-245482.html?legacy=cnet</a>&quot;&gt;<a href="http://news.com.com/2100-1023-245482.html?legacy=cnet">http://news.com.com/2100-1023-245482.html?legacy=cnet</a>&lt;/a&gt; 
&lt;a href=&quot;<a href="http://www.cnn.com/2000/TECH/computing/01/21/real.streambox.idg/">http://www.cnn.com/2000/TECH/computing/01/21/real.streambox.idg/</a>&quot;&gt;<a href="http://www.cnn.com/2000/TECH/computing/01/21/real.streambox.idg/">http://www.cnn.com/2000/TECH/computing/01/21/real.streambox.idg/</a>&lt;/a&gt; 
 
In essence, what it allowed you to do was save files by circumventing
the Realnetworks protection scheme and cache streamed content.  It
accomplished this by mimicking the Realnetworks RTSP spec, fooling the
server into thinking it was delivering a normal stream.  However, the
Streambox program would process the file, but would also convert the
streamed data into a format that could be saved.  This was still
entirely dependent on the streamed quality, which would mean
jitters/freezes would occur in the saved copy if problems were
encountered during the stream.
 
Duplicating such actions today may result in similar court action seen
in 2000 under the Digital Millennium Copyright Act which is why you
don’t see any products taking Streambox’s place.  However, there are
numerous plugins available for player programs that can interpret
incoming streams and simultaneously convert and save them in other
formats.  The legality of such plugins is questionable, which is why I
will not post links.
&lt;a href=&quot;<a href="http://www.loc.gov/copyright/legislation/dmca.pdf">http://www.loc.gov/copyright/legislation/dmca.pdf</a>&quot;&gt;<a href="http://www.loc.gov/copyright/legislation/dmca.pdf">http://www.loc.gov/copyright/legislation/dmca.pdf</a>&lt;/a&gt; 
 
While I realize I have omitted numerous data specs, I have attempted
to address the original intent of your question.  If I have not made
certain items clear, please let me know and I’ll try and clear up any
misunderstanding.
 
Thanks, 
carwfloc-ga  

---

Request for Answer Clarification by arjen-ga on 28 May 2002 17:50 PDT

The document about the RTSP is meant for firewall/proxy developers. It
is not usefull for writing a client as it misses the response for:
RealChallenge1: 7a4985d28a995e345af2a60b8050a06c 
 
About the streambox case: They where found guilty for not implementing
the copy protection bit. Being compatible with realplayer is not
illegal.  

---

Clarification of Answer by carwfloc-ga on 28 May 2002 19:14 PDT

Hello arjen, 
 
Thanks for the clarification of your original question.  I apologize
for misinterpreting the intent of your question as I attempted to
answer it with information on how Realplayer works versus Streambox,
and not a question about debugging a specific client response.
 
I have not programmed any Realnetworks programs, but I did download a
couple of SDKs for Realsystem and have tried to understand the
client/server communication a little (they provide quite good
documentation).  It is my understanding that you're coding a Real
client, and wish it to be able to respond to the server's
RealChallenge request.  If so, I would suggest posting your problem to
the Realnetworks Developer's forum
[&lt;a href=&quot;<a href="http://realforum.real.com/cgi-bin/realforum/wwwthreads.pl?Cat=">http://realforum.real.com/cgi-bin/realforum/wwwthreads.pl?Cat=</a>&quot;&gt;<a href="http://realforum.real.com/cgi-bin/realforum/wwwthreads.pl?Cat=">http://realforum.real.com/cgi-bin/realforum/wwwthreads.pl?Cat=</a>&lt;/a&gt;] or to
an appropriate newsgroup, such as comp.multimedia.
 
If you'd like to repost it to Google Answers, may I suggest reposting
this question with a subject line similar to:  &amp;quot;Looking for handshake
header calculation of RealChallenge request IDs in a user-built
Realserver.&amp;quot;  You could go further by posting the complete packets you
have intercepted between your client and Realserver, since calculation
of the ClientChallenge/RealChallenge is probably not based on the SDP
protocol decisions being carried out in the client/server responses. 
I have a feeling that the ClientChallenge/RealChallenge
authentications are uniquely generated due to Realnetworks security.
 
As for the Streambox ruling, I am aware of the outcomes of the
lawsuit, among them being the court's dislike of Streambox's &amp;quot;hack&amp;quot; of
the Realplayer code.  Note that I did not discuss the legal
proceedings in my answer.
 
Good luck in your coding, and please clarify if you'd like to
repost/request a refund.
carwfloc-ga  

---

Clarification of Answer by carwfloc-ga on 29 May 2002 22:04 PDT

Hello arjen, 
 
I am disappointed that you have found my answer to your original
question unsatisfactory.  Please request a refund and I will be happy
to pass a note along to the Google Answers Editors to expedite this
process.  I also urge you to repost your question so a Google
Researcher who has also happened to have coded a Realplayer RTSP
client program can address your specific requirements.
 
If you do decide to repost your question may I suggest (for future
reference) reviewing the Google Answers Helps and Tips section on
writing a good question.  The first section explains that to "get a
great answer" you should "ask a great question" with some tips to
specify your question:
<a href="https://answers.google.com/answers/help.html#howtospecify">https://answers.google.com/answers/help.html#howtospecify</a> 
 
Thank you for using Google Answers and I wish you good luck in your
programming.
carwfloc-ga 

The information is useless. I asked for information about how to
download files, and I got info on how to write a proxy. That
information was not enough, as it missed required secret parts (the
realchallenge handshake.)

The information that was given, was impressive though.

With the information I got, I can't download a single file.

The protocol itself is not documented, and they will presumably not
like you attempting to reverse engineer it.  (plus, they've had enough
time to make it pretty hard by now.)

The RealSystem SDK, which you can download at
http://forms.real.com/rnforms/resources/server/realsystemsdk/index.html,
should have everything you need to save local copies of files; to
download it, you need to agree to a license agreement which, among
other things, requires you to respect the copy protection bits of any
content, but does seem to allow the possibility of saving local files.
 (though it places many other restrictions on what you can do, and in
any case, I'm certainly not a lawyer.)

Two samples you might look at within the SDK are "testplay" and
"pcmrendr" which together would allow you to write a custom player
which would extract the audio from a realaudio stream....

of course, I don't know whether StreamBox used the Real SDK or whether
they implemented the protocol themselves; most likely, they used the
SDK, albeit an earlier version than currently available.

Hope this is helpful for you; if not, well, what can you expect for
the price.

Thanks for using google answers!

The question was: "What is the protocol..."

Answer: "...not documented...  ...Hope this is helpful for you; if
not, well, what can you expect for the price."

That is a comment, not an answer. If the price is too low, leave the
question open or leave a comment. If the answer is unknown, leave the
question open.

The link in the answer is useless, as it points to an SDK for writing
plugins. Realplayer needs to be installed for that.

I did put the time frame at 1 year. I got two useless answers in three
days. Is it that hard to leave a question open? Maybe there are too
many researchers per question.

P.S. My question page is quite a mess now. The repost request is not
'publically viewable' at the moment and there is no 'Request Answer
Clarification' button at the second answer. I think the comments that
are posted before the second answer should be above the second answer.

To summarise: The entire intention of the protocol is to prevent you
from downloading a single file.  To download a file which is protected
by the system is wrong.  Moreover, the Answer above answers your
Question, which was 'what is the protocol used?'. But it looks like
you have rejected the answer to your question because you failed to
bypass a copy protection system.

No, the intention of the protocol is the protection of their broken
copy protection. There is no real copy protection. AFTER you
downloaded a single file you can check for the copy protection bit.
When that is set the client should deny saving the file, and only
allow streaming the file. I don't mind implementing that part of the
spec. But first I need the protocol. StreamBox found it. If it is
documented somewhere, I'd like to pay for it.