<jonbrwn>
this particular bit is demo value generator, you'd need some meaningful data for Wallaroo to ingest
<zodiac403>
Do I have to implement this code in my app or can I make Wallaroo call this?
<zodiac403>
> you'd need some meaningful data for Wallaroo to ingest
<zodiac403>
I have a script that is publishing messages on a given Topic on Redis.
<zodiac403>
But that currently is faaar separated from the Wallaroo processes...
<jonbrwn>
10:21 AM <zodiac403> I have a script that is publishing messages on a given Topic on Redis.
<jonbrwn>
are the messages in a format that the alerts_stateless application can process or is it for a different application?
<zodiac403>
Format is a simple string "foobar".
<zodiac403>
yet
<zodiac403>
<zodiac403> Do I have to implement this code (the redis subscribe) in my app or can I make Wallaroo call this?
<jonbrwn>
the code that generates data should be implemented outside of Wallaroo
<zodiac403>
Sorry, I'll be offline in a few minutes to catch my train...
<jonbrwn>
no worries, did that at least help in getting that bit of the Redis setup explained?
<jonbrwn>
either way, do feel free to ping us when you're available so we can help.
<zodiac403>
Not yet... How can I tell the Wallaroo Pipeline to actually call the redis_subscriber_source and e.g. pass on my TOPIC to subscribe to?
<zodiac403>
Did I miss an important part in your documentation?
<jonbrwn>
it's very possible that we didn't describe it in the best way for users to consume
<zodiac403>
Thx, Jonathan so far. I'll try to come back later...
<zodiac403>
;)
zodiac403 has quit [Quit: Page closed]
zodiac403 has joined #wallaroo
<zodiac403>
Hi @jonbrwn, are you still there?
<jonbrwn>
yes @zodiac403
<jonbrwn>
if you wanted to continue to use the `alerts_stateless` application, I'd recommend setting up an external script which uses the generator code that I linked to above to publish to your Redis topic
<jonbrwn>
then we can get you setup with the Redis connector source
<jonbrwn>
you'll end up having 2 separate processes: one for the redis source connector and one for the wallaroo application
<jonbrwn>
you will also need a Redis Sink Connector process if you plan to write out to Redis from Wallaroo as well
<zodiac403>
> "one for the redis source connector and one for the wallaroo application"
<zodiac403>
How do these 2 communicate with eachother?
<jonbrwn>
when you pass the application_module to the connector, it uses the port defined in the SourceConnectorConfig to send the data over to the running Wallaroo application using the same module
<zodiac403>
ok. give me some minutes...
<zodiac403>
;)
<jonbrwn>
no worries, I'll check back in a few
zodiac403 has quit [Ping timeout: 256 seconds]
zodiac403 has joined #wallaroo
<zodiac403>
Hi @jonbrwn, sorry, got a little distracted.
<zodiac403>
Now I have alerts.py and redis_subscriber_source.py in my working directory.
<jonbrwn>
what's in your redis_subscriber_source.py file?
<zodiac403>
I try to start it with the following line but that doesn't feel right. ```python redis_client.py --application-module alerts --connector c --c-topic mytopic --out 127.0.0.1:5555 --metrics 127.0.0.1:5001 --control 127.0.0.1:6000 --data 127.0.0.1:6001 --name worker-name --external 127.0.0.1:5050 --cluster-initializer --ponythreads=1 --ponynobl```
<jonbrwn>
so you'd run `python redis_subscriber_source ...` instead of `redis_client.py`
<jonbrwn>
and a few of those args look off
<jonbrwn>
give me a few
<zodiac403>
argh, yeah copy-paste-error in the chat
<strmpnk>
A note on the command line arguments: We usually pass the extra arguments so that your application can parse out the same fields when loaded from either the connector script or the wallaroo worker. It's worth noting you can leave many of those off like --name --data --control --external and so on. You do need --application-module, --connector, and any connector required arguments (as well as any your application script expects
<strmpnk>
to parse).
<strmpnk>
The --c- prefix, in this case, is there to avoid any command line argument name conflicts between the connector instance specific parameters and wallaroo ones. TL/DR, if you're not sure, it's safe to pass all of the wallaroo arguments through.
<jonbrwn>
@zodiac403 I'm stepping out for a few, @strmkpnk will help you while I'm gone
<strmpnk>
zodiac403: I wrote the original connector script you're using so you can blame me if it's unclear. ;-) I'm happy to help port over the code.
<zodiac403>
thx ;)
<zodiac403>
As you can read above, I have the `alerts_stateless` sample up and running. But I feel a little lost connecting it to a Redis subscriber.
<zodiac403>
If you have a working example or a piece of docs I missed so far, I'd be happy.
<strmpnk>
Okay. So let's look at your application module. Can you paste it in a gist or pastebin? We can walk through that first.
<strmpnk>
Or I can start from the sample and we can do it together if you prefer.
<strmpnk>
Okay. It looks good. There is a minor note, depending on which version of python/machida you're running. The decoder is returning the byte-string directly. So if you're expecting a regular python string, you might want to decode that.
<strmpnk>
The main problem, which I think is the error is the port number.
<strmpnk>
--data defines an internal communication port for data between workers. It's possibly too vague of a name.
<strmpnk>
If you change that port in the source configuration to something like 7000 it should be able to listen for connections properly.
<zodiac403>
Which port do you mean?
<strmpnk>
Line 37 in the gist.
<strmpnk>
It's a connector instance specific port, which should be separate from the rest.
<strmpnk>
If you run it, it should show that it can connect (not sure it will work just yet since we need to parse data into a working format for the example and this depends on how you put things into redis).
<zodiac403>
OK. I changed the port. But how do I start it?
<zodiac403>
Is `python redis_subscriber_source.py` the correct entry point?
<strmpnk>
Okay. So you can start both processes in any order but I usually start the worker first. You can run the worker with the same command line as the example's readme first.
<zodiac403>
Yes, I have that "... Alerts (stateless) source attempting to listen on 127.0.0.1:7000 ..."
<strmpnk>
Once that is ready, we can run the script. This should need three arguments: --application-module, --connector, and --c-topic
<strmpnk>
The application module must be in the python path so it can load it and use the proper encoder and discover the port to use.
<strmpnk>
The --connector name should match what you use in the application module, so "c" like you have.
<strmpnk>
And then the connector specific topic with the prefix.
<zodiac403>
Wait, I get a python error...
<strmpnk>
What does it say?
<zodiac403>
I call `python redis_subscriber_source.py --application-module alerts --connector c --c-topic foo`
<strmpnk>
Great. So we're not completely set just yet but before we move on, I can explain any confusion over which arguments were passed and why.
<zodiac403>
OK, I'm back. Currently I call the redis script with `python redis_subscriber_source.py --application-module alerts --connector c --c-topic foo --c-host localhost --c-port 6379 --out 127.0.0.1:5555`
<zodiac403>
Now I get some redis connection issues, but that seems to be part of my docker setup.
<zodiac403>
<@strmpnk> ... I can explain any confusion over which arguments were passed and why. >>> Thanks, but that is understandable for me.
<strmpnk>
Ah. Yeah, docker may not be mapping the redis port to localhost. `-p` to the docker command line might help (or equivalent if you're using other tools to setup the container)
<strmpnk>
So the next step is to use the encode_feed call to construct Transaction objects and then adjust decode_feed to turn the bytes back into a Python object.
<strmpnk>
We have many options here. To illustrate with text: [something that writes to topic]-->[redis]-->[connector (encoder)--]--[-->(decoder) wallaroo]-->[sink]
<strmpnk>
We can setup whatever format we want as long as (decoder) can turn it into a Transaction object.
<strmpnk>
So redis could use a specific format (JSON, pickled object, text format, something custom).
<strmpnk>
But in this example Transaction has two parts: the transaction id and the amount. So we can pass those however we want.
<strmpnk>
(encode)'s job is to take whatever is in redis and do any translation it need to for (decode) to do it's job.
<strmpnk>
It may not need to do anything, or it could filter/clean up the data. Up to you.
<zodiac403>
cool.
<zodiac403>
Still having some redis connection issues on this side. But that I'll figure out myself.
<strmpnk>
In your example, you have decode passing something from struct. Here, we'd need to construct a Transaction.
<zodiac403>
When the publisher triggers the encoder, which payload type (string / binary / ?) will I expect?
<zodiac403>
DECODER of course...
<strmpnk>
The encoder gets whatever write() passes. The decoder gets the same byte string that encoder returns for each item.
<strmpnk>
(Given that this is all being passed over TCP)