Ed: solved with the help of the async_stream crate.

I’m struggling with the borrow checker!

My problem: I’m using actix-web and rusqlite. I want to return an unlimited number of records from an rusqlite query, and actix provides a Stream trait for that kind of thing. You just impl the trait and return your records from a poll_next() fn.

On the rusqlite side, there’s this query_map that returns an iterator of records from a query. All I have to do is smush these two features together.

So the plan is to put the iterator returned by query_map into a struct that impls Stream. Problem is the lifetime of a var used by query_map. How to make the var have the same lifetime as the iterator??

So here’s the code:

pub struct ZkNoteStream<'a, T> {
  rec_iter: Box<dyn Iterator<Item = T> + 'a>,
}

// impl of Stream just calls next() on the iterator.  This compiles fine.
impl<'a> Stream for ZkNoteStream<'a, serde_json::Value> {
  type Item = serde_json::Value;

  fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
    Poll::Ready(self.rec_iter.next())
  }
}

// init function to set up the ZkNoteStream.
impl<'a> ZkNoteStream<'a, Result<ZkListNote, rusqlite::Error>> {
  pub fn init(
    conn: &'a Connection,
    user: i64,
    search: &ZkNoteSearch,
  ) -> Result<Self, Box<dyn Error>> {
    let (sql, args) = build_sql(&conn, user, search.clone())?;

    let sysid = user_id(&conn, "system")?;
    let mut pstmt = conn.prepare(sql.as_str())?;

    // Here's the problem!  Borrowing pstmt.
    let rec_iter = pstmt.query_map(rusqlite::params_from_iter(args.iter()), move |row| {
      let id = row.get(0)?;
      let sysids = get_sysids(&conn, sysid, id)?;
      Ok(ZkListNote {
        id: id,
        title: row.get(1)?,
        is_file: {
          let wat: Option<i64> = row.get(2)?;
          wat.is_some()
        },
        user: row.get(3)?,
        createdate: row.get(4)?,
        changeddate: row.get(5)?,
        sysids: sysids,
      })
    })?;

    Ok(ZkNoteStream::<Result<ZkListNote, rusqlite::Error>> {
      rec_iter: Box::new(rec_iter),
    })
  }
}

And here’s the error:

error[E0515]: cannot return value referencing local variable `pstmt`
   --> server-lib/src/search.rs:170:5
    |
153 |       let rec_iter = pstmt.query_map(rusqlite::params_from_iter(args.iter()), move |row| {
    |                      ----- `pstmt` is borrowed here
...
170 | /     Ok(ZkNoteStream::<Result<ZkListNote, rusqlite::Error>> {
171 | |       rec_iter: Box::new(rec_iter),
172 | |     })
    | |______^ returns a value referencing data owned by the current function

So basically it boils down to pstmt getting borrowed in the query_map call. It needs to have the same lifetime as the closure. How do I ensure that?

  • pr06lefs@lemmy.mlOP
    link
    fedilink
    arrow-up
    1
    ·
    7 months ago

    UPDATE!

    I sort of solved this part of it, or at least got it to compile. I’ve got a reddit post of this too! Someone there hinted that I should use another struct ‘above’ ZkNoteStream. I’m doing that in the code listing below.

    ZnsMaker has an init() fn, then you call make_stream() and it returns a ZkNoteStream. The intent is ZnsMaker should be managed so it lasts as long as the ZkNoteStream needs to last. All this bit compiles, great! But when I go to use it in my actix handler, I get a borrowing problem there instead. So I may have just kicked the can down the road.

    This part compiles. Wrong types still, should produce Bytes instead of ZkListNotes.

    pub struct ZkNoteStream&lt;'a, T> {
      rec_iter: Box + 'a>,
    }
    
    impl&lt;'a> Stream for ZkNoteStream&lt;'a, Result> {
      type Item = Result;
    
      fn poll_next(mut self: Pin&lt;&amp;mut Self>, cx: &amp;mut Context&lt;'_>) -> Poll> {
        Poll::Ready(self.rec_iter.next())
      }
    }
    
    pub struct ZnsMaker&lt;'a> {
      pstmt: rusqlite::Statement&lt;'a>,
      sysid: i64,
      args: Vec,
    }
    
    impl&lt;'a> ZnsMaker&lt;'a> {
      pub fn init(
        conn: &amp;'a Connection,
        user: i64,
        search: &amp;ZkNoteSearch,
      ) -> Result> {
        let (sql, args) = build_sql(&amp;conn, user, search.clone())?;
    
        let sysid = user_id(&amp;conn, "system")?;
    
        Ok(ZnsMaker {
          args: args,
          sysid: sysid,
          pstmt: conn.prepare(sql.as_str())?,
        })
      }
    
      pub fn make_stream(
        &amp;'a mut self,
        conn: &amp;'a Connection,  // have to pass the connection in here instead of storing in ZnsMaker, for Reasons.
      ) -> Result>, rusqlite::Error> {
        let sysid = self.sysid;
        let rec_iter =
          self
            .pstmt
            .query_map(rusqlite::params_from_iter(self.args.iter()), move |row| {
              let id = row.get(0)?;
              let sysids = get_sysids(&amp;conn, sysid, id)?;
              Ok(ZkListNote {
                id: id,
                title: row.get(1)?,
                is_file: {
                  let wat: Option = row.get(2)?;
                  wat.is_some()
                },
                user: row.get(3)?,
                createdate: row.get(4)?,
                changeddate: row.get(5)?,
                sysids: sysids,
              })
            })?;
    
        Ok(ZkNoteStream::&lt;'a, Result> {
          rec_iter: Box::new(rec_iter),
        })
      }
    }
    

    Ok and here’s the handler function where I receive a query and make the ZnsMaker. But if I create a ZkNoteStream with it, I get a borrowing error. Maybe it would be ok if I immediately consumed it in an HttpResponse::Ok().streaming(znsstream). Got to fix the types first though.

    pub async fn zk_interface_loggedin_streaming(
      config: &amp;Config,
      uid: i64,
      msg: &amp;UserMessage,
    ) -> Result> {
      match msg.what.as_str() {
        "searchzknotesstream" => {
          let msgdata = Option::ok_or(msg.data.as_ref(), "malformed json data")?;
          let search: ZkNoteSearch = serde_json::from_value(msgdata.clone())?;
          let conn = sqldata::connection_open(config.orgauth_config.db.as_path())?;
          let mut znsm = ZnsMaker::init(&amp;conn, uid, &amp;search)?;
          {
            // borrowed value of znsm doesn't live long enough!  wat do?
            let znsstream = &amp;znsm.make_stream(&amp;conn)?;
          }
          Err("wat".into())
        }
        wat => Err(format!("invalid 'what' code:'{}'", wat).into()),
      }
    }
    
    • hallettj@beehaw.org
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      7 months ago

      I’m glad you found a workaround. I didn’t want to be defeated by the callback idea that I had yesterday so I worked on it some more and came up with a similar-but-different solution where ZnsMaker stores pstmt paired with a closure that borrows it. This is again a simplified version of your code that is self-contained:

      https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2021&amp;gist=5bd6fb7c1cbf1c9c44c8f4bdbb1e8074

      The closure solution avoids the need to pass conn to make_stream. I don’t know if it would fix the new borrowing error that you got since I did not reproduce that error. But I think it might.

      Does znsstream need to outlive znsm? If so I think my solution solves the problem. Does znsstream need to outlive conn? That would be more complicated.

      Edit: Oh! You don’t need to put both the pstmt and a closure in ZnsMaker. Instead you can just store a closure if you move ownership of pstmt into the closure. That ensures that pstmt lives as long as the function that produces the iterator that you want:

      https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2021&amp;gist=491258a7dc9bcad9dab08632d68c026b

      Edit: Lol you don’t need ZnsMaker after all. See my other top-level comment. I learned some things working on this, so time well spent IMO.