Multi message storage (#8182)

* First try at multimessage storage and display * Nrf built issue fix * Message view mode * Add channel name instead of channel slot * trunk fix * Fix for DM threading * fix for message time * rename of view mode to Conversations * Reply in thread feature * rename Select View Mode to Select Conversation * dismiss all live fix * Messages from phone show on screen * Decoupled message packets from screen.cpp and cleaned up * Cannedmessage cleanup and emotes fixed * Ack on messages sent * Ack message cleanup * Dismiss feature fixed * removed legacy temporary messages * Emote picker fix * Memory size debug * Build error fix * Sanity checks are okay sometimes * Lengthen channel name and finalize cleanup removal of Broadcast * Change DM to @ in order to unify on a single method * Continue unifying display, also show message status on the "isMine" lines * Add context for incoming messages * Better to say "in" vs "on" * crash fix for confirmation nodes * Fix outbound labels based to avoid creating delays * Eink autoscroll dissabled * gating for message storage when not using a screen * revert * Build fail fix * Don't error out with unset MAC address in unit tests * Provide some extra spacing for low hanging characters in messages * Reorder menu options and reword Respond * Reword menus to better reflect actions * Go to thread from favorite screen * Reorder Favorite Action Menu with simple word modifications * Consolidate wording on "Chats" * Mute channel fix * trunk fix * Clean up how muting works along with when we wake the screen * Fix builds for HELTEC_MESH_SOLAR * Signal bars for message ack * fix for notification renderer * Remove duplicate code, fix more Chats, and fix C6L MessageRenderer * Fix to many warnings related to BaseUI * preset aware signal strength display * More C6L fixes and clean up header lines * Use text aligns for message layout where necessary * Attempt to fix memory usage of invalidLifetime * Update channel mute for adjusted protobuf * Missed a comma in merge conflicts * cleanup to get more space * Trunk fixes * Optimize Hi Rez Chirpy to save space * more fixes * More cleanup * Remove used getConversationWith * Remove unused dismissNewestMessage * Fix another build error on occassion * Dimiss key combo function deprecated * More cleanup * Fn symbol code removed * Waypoint cleanup * Trunk fix * Fixup Waypoint screen with BaseUI code * Implement Haruki's ClockRenderer and broadcast decomposeTime across various files. * Revert "Implement Haruki's ClockRenderer and broadcast decomposeTime across various files." This reverts commit 2f65721774. * Implement Haruki's ClockRenderer and broadcast decomposeTime across various files. Attempt 2! * remove memory usage debug * Revert only RangeTestModule.cpp change * Switch from dynamic std::string storage to fixed-size char[] * Removing old left over code * More optimization * Free Heap when not on Message screen * build error fixes * Restore ellipsis to end of long names * Remove legacy function renderMessageContent * improved destination filtering * force PKI * cleanup * Shorten longNames to not exceed message popups * log messages sent from apps * Trunk fix * Improve layout of messages screen * Fix potential crash for undefined variable * Revert changes to RedirectablePrint.cpp * Apply shortening to longNames in Select Destination * Fix short name displays * Fix sprintfOverlappingData issue * Fix nullPointerRedundantCheck warning on ESP32 * Add "Delete All Chats" to all chat views * Improve getSafeNodeName / sanitizeString code. * Improve getSafeNodeName further * Restore auto favorite; but only if not CLIENT_BASE * Don't favorite if WE are CLIENT_BASE role * Don't run message persistent in MUI * Fix broken endifs * Unkwnown nodes no longer show as ??? on message thread * More delete options and cleanup of code * fix for delete this chat * Message menu cleanup * trunk fix * Clean up some menu options and remove some Unit C6L ifdefines * Rework Delete flow * Desperate times call for desperate measures * Create a background on the connected icon to reduce overlap impact * Optimize code for background image * Fix for Muzi_Base * Trunk Fixes * Remove the up/down shortcut to launch canned messages (#8370) * Remove the up/down shortcut to launch canned messages * Enabled MQTT and WEBSERVER by default (#8679) Signed-off-by: kur1k0 <zhuzirun@m5stack.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> Co-authored-by: Jonathan Bennett <jbennett@incomsystems.biz> --------- Signed-off-by: kur1k0 <zhuzirun@m5stack.com> Co-authored-by: Riker <zhuzirun@m5stack.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> * Correct string length calculation for signal bars * Manual message scrolling * Fix * Restore CannedMessages on Home Frame * UpDown situational destination for textMessage * Correct up/down destinations on textMessage frame * Update Screen.h for handleTextMessage * Update Screen.cpp to repair a merge issue * Add nudge scroll on UpDownEncoder devices. * Set nodeName to maximum size * Revert "Set nodeName to maximum size" This reverts commit e254f39925. * Reflow Node Lists and TLora Pager Views (#8942) * Add files via upload * Move files into the right place * Short or Long Names for everyone! * Add scrolling to Node list * Pagination fix for Latest to oldest per page * Page counters * Dynamic scaling of column counts based upon screen size, clean up box drawing * Reflow Node Lists and TLora Pager Views (#8942) * Add files via upload * Move files into the right place * Short or Long Names for everyone! * Add scrolling to Node list * Pagination fix for Latest to oldest per page * Page counters * Dynamic scaling of column counts based upon screen size, clean up box drawing * Update exempt labels for stale bot workflow Adds triaged and backlog to the list of exempt labels. * Update naming of Frame Visibility toggles * Fix to scrolling * Fix for content cutting off when from us * Fix for "delete this chat" now it does delete the current one * Rework isHighResolution to be an enum called ScreenResolution * Migrate Unit C6L macro guards into currentResolution UltraLow checks * Mistakes happen - restoring NodeList Renderer line --------- Signed-off-by: kur1k0 <zhuzirun@m5stack.com> Co-authored-by: Jason P <applewiz@mac.com> Co-authored-by: Jonathan Bennett <jbennett@incomsystems.biz> Co-authored-by: Riker <zhuzirun@m5stack.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> Co-authored-by: whywilson <m.tools@qq.com> Co-authored-by: Tom Fifield <tom@tomfifield.net>
2025-12-29 14:10:53 +00:00 · 2025-12-24 17:13:31 -05:00
parent e5c3eda2a2
commit 9da4396c6f
37 changed files with 3337 additions and 1656 deletions
--- a/src/MessageStore.cpp
+++ b/src/MessageStore.cpp
@@ -0,0 +1,426 @@
+#include "configuration.h"
+#if HAS_SCREEN
+#include "FSCommon.h"
+#include "MessageStore.h"
+#include "NodeDB.h"
+#include "SPILock.h"
+#include "SafeFile.h"
+#include "gps/RTC.h"
+#include "graphics/draw/MessageRenderer.h"
+#include <cstring> // memcpy
+
+#ifndef MESSAGE_TEXT_POOL_SIZE
+#define MESSAGE_TEXT_POOL_SIZE (MAX_MESSAGES_SAVED * MAX_MESSAGE_SIZE)
+#endif
+
+// Global message text pool and state
+static char *g_messagePool = nullptr;
+static size_t g_poolWritePos = 0;
+
+// Reset pool (called on boot or clear)
+static inline void resetMessagePool()
+{
+    if (!g_messagePool) {
+        g_messagePool = static_cast<char *>(malloc(MESSAGE_TEXT_POOL_SIZE));
+        if (!g_messagePool) {
+            LOG_ERROR("MessageStore: Failed to allocate %d bytes for message pool", MESSAGE_TEXT_POOL_SIZE);
+            return;
+        }
+    }
+    g_poolWritePos = 0;
+    memset(g_messagePool, 0, MESSAGE_TEXT_POOL_SIZE);
+}
+
+// Allocate text in pool and return offset
+// If not enough space remains, wrap around (ring buffer style)
+static inline uint16_t storeTextInPool(const char *src, size_t len)
+{
+    if (len >= MAX_MESSAGE_SIZE)
+        len = MAX_MESSAGE_SIZE - 1;
+
+    // Wrap pool if out of space
+    if (g_poolWritePos + len + 1 >= MESSAGE_TEXT_POOL_SIZE) {
+        g_poolWritePos = 0;
+    }
+
+    uint16_t offset = g_poolWritePos;
+    memcpy(&g_messagePool[g_poolWritePos], src, len);
+    g_messagePool[g_poolWritePos + len] = '\0';
+    g_poolWritePos += (len + 1);
+    return offset;
+}
+
+// Retrieve a const pointer to message text by offset
+static inline const char *getTextFromPool(uint16_t offset)
+{
+    if (!g_messagePool || offset >= MESSAGE_TEXT_POOL_SIZE)
+        return "";
+    return &g_messagePool[offset];
+}
+
+// Helper: assign a timestamp (RTC if available, else boot-relative)
+static inline void assignTimestamp(StoredMessage &sm)
+{
+    uint32_t nowSecs = getValidTime(RTCQuality::RTCQualityDevice, true);
+    if (nowSecs) {
+        sm.timestamp = nowSecs;
+        sm.isBootRelative = false;
+    } else {
+        sm.timestamp = millis() / 1000;
+        sm.isBootRelative = true;
+    }
+}
+
+// Generic push with cap (used by live + persisted queues)
+template <typename T> static inline void pushWithLimit(std::deque<T> &queue, const T &msg)
+{
+    if (queue.size() >= MAX_MESSAGES_SAVED)
+        queue.pop_front();
+    queue.push_back(msg);
+}
+
+template <typename T> static inline void pushWithLimit(std::deque<T> &queue, T &&msg)
+{
+    if (queue.size() >= MAX_MESSAGES_SAVED)
+        queue.pop_front();
+    queue.emplace_back(std::move(msg));
+}
+
+MessageStore::MessageStore(const std::string &label)
+{
+    filename = "/Messages_" + label + ".msgs";
+    resetMessagePool(); // initialize text pool on boot
+}
+
+// Live message handling (RAM only)
+void MessageStore::addLiveMessage(StoredMessage &&msg)
+{
+    pushWithLimit(liveMessages, std::move(msg));
+}
+void MessageStore::addLiveMessage(const StoredMessage &msg)
+{
+    pushWithLimit(liveMessages, msg);
+}
+
+// Add from incoming/outgoing packet
+const StoredMessage &MessageStore::addFromPacket(const meshtastic_MeshPacket &packet)
+{
+    StoredMessage sm;
+    assignTimestamp(sm);
+    sm.channelIndex = packet.channel;
+
+    const char *payload = reinterpret_cast<const char *>(packet.decoded.payload.bytes);
+    size_t len = strnlen(payload, MAX_MESSAGE_SIZE - 1);
+    sm.textOffset = storeTextInPool(payload, len);
+    sm.textLength = len;
+
+    // Determine sender
+    uint32_t localNode = nodeDB->getNodeNum();
+    sm.sender = (packet.from == 0) ? localNode : packet.from;
+
+    sm.dest = packet.to;
+
+    bool isDM = (sm.dest != 0 && sm.dest != NODENUM_BROADCAST);
+
+    if (packet.from == 0) {
+        sm.type = isDM ? MessageType::DM_TO_US : MessageType::BROADCAST;
+        sm.ackStatus = AckStatus::NONE;
+    } else {
+        sm.type = isDM ? MessageType::DM_TO_US : MessageType::BROADCAST;
+        sm.ackStatus = AckStatus::ACKED;
+    }
+
+    addLiveMessage(sm);
+    return liveMessages.back();
+}
+
+// Outgoing/manual message
+void MessageStore::addFromString(uint32_t sender, uint8_t channelIndex, const std::string &text)
+{
+    StoredMessage sm;
+
+    // Always use our local time (helper handles RTC vs boot time)
+    assignTimestamp(sm);
+
+    sm.sender = sender;
+    sm.channelIndex = channelIndex;
+    sm.textOffset = storeTextInPool(text.c_str(), text.size());
+    sm.textLength = text.size();
+
+    // Use the provided destination
+    sm.dest = sender;
+    sm.type = MessageType::DM_TO_US;
+
+    // Outgoing messages always start with unknown ack status
+    sm.ackStatus = AckStatus::NONE;
+
+    addLiveMessage(sm);
+}
+
+#if ENABLE_MESSAGE_PERSISTENCE
+
+// Compact, fixed-size on-flash representation using offset + length
+struct __attribute__((packed)) StoredMessageRecord {
+    uint32_t timestamp;
+    uint32_t sender;
+    uint8_t channelIndex;
+    uint32_t dest;
+    uint8_t isBootRelative;
+    uint8_t ackStatus;           // static_cast<uint8_t>(AckStatus)
+    uint8_t type;                // static_cast<uint8_t>(MessageType)
+    uint16_t textLength;         // message length
+    char text[MAX_MESSAGE_SIZE]; // store actual text here
+};
+
+// Serialize one StoredMessage to flash
+static inline void writeMessageRecord(SafeFile &f, const StoredMessage &m)
+{
+    StoredMessageRecord rec = {};
+    rec.timestamp = m.timestamp;
+    rec.sender = m.sender;
+    rec.channelIndex = m.channelIndex;
+    rec.dest = m.dest;
+    rec.isBootRelative = m.isBootRelative;
+    rec.ackStatus = static_cast<uint8_t>(m.ackStatus);
+    rec.type = static_cast<uint8_t>(m.type);
+    rec.textLength = m.textLength;
+
+    // Copy the actual text into the record from RAM pool
+    const char *txt = getTextFromPool(m.textOffset);
+    strncpy(rec.text, txt, MAX_MESSAGE_SIZE - 1);
+    rec.text[MAX_MESSAGE_SIZE - 1] = '\0';
+
+    f.write(reinterpret_cast<const uint8_t *>(&rec), sizeof(rec));
+}
+
+// Deserialize one StoredMessage from flash; returns false on short read
+static inline bool readMessageRecord(File &f, StoredMessage &m)
+{
+    StoredMessageRecord rec = {};
+    if (f.readBytes(reinterpret_cast<char *>(&rec), sizeof(rec)) != sizeof(rec))
+        return false;
+
+    m.timestamp = rec.timestamp;
+    m.sender = rec.sender;
+    m.channelIndex = rec.channelIndex;
+    m.dest = rec.dest;
+    m.isBootRelative = rec.isBootRelative;
+    m.ackStatus = static_cast<AckStatus>(rec.ackStatus);
+    m.type = static_cast<MessageType>(rec.type);
+    m.textLength = rec.textLength;
+
+    // 💡 Re-store text into pool and update offset
+    m.textLength = strnlen(rec.text, MAX_MESSAGE_SIZE - 1);
+    m.textOffset = storeTextInPool(rec.text, m.textLength);
+
+    return true;
+}
+
+void MessageStore::saveToFlash()
+{
+#ifdef FSCom
+    // Ensure root exists
+    spiLock->lock();
+    FSCom.mkdir("/");
+    spiLock->unlock();
+
+    SafeFile f(filename.c_str(), false);
+
+    spiLock->lock();
+    uint8_t count = static_cast<uint8_t>(liveMessages.size());
+    if (count > MAX_MESSAGES_SAVED)
+        count = MAX_MESSAGES_SAVED;
+    f.write(&count, 1);
+
+    for (uint8_t i = 0; i < count; ++i) {
+        writeMessageRecord(f, liveMessages[i]);
+    }
+    spiLock->unlock();
+
+    f.close();
+#endif
+}
+
+void MessageStore::loadFromFlash()
+{
+    std::deque<StoredMessage>().swap(liveMessages);
+    resetMessagePool(); // reset pool when loading
+
+#ifdef FSCom
+    concurrency::LockGuard guard(spiLock);
+
+    if (!FSCom.exists(filename.c_str()))
+        return;
+
+    auto f = FSCom.open(filename.c_str(), FILE_O_READ);
+    if (!f)
+        return;
+
+    uint8_t count = 0;
+    f.readBytes(reinterpret_cast<char *>(&count), 1);
+    if (count > MAX_MESSAGES_SAVED)
+        count = MAX_MESSAGES_SAVED;
+
+    for (uint8_t i = 0; i < count; ++i) {
+        StoredMessage m;
+        if (!readMessageRecord(f, m))
+            break;
+        liveMessages.push_back(m);
+    }
+
+    f.close();
+#endif
+}
+
+#else
+// If persistence is disabled, these functions become no-ops
+void MessageStore::saveToFlash() {}
+void MessageStore::loadFromFlash() {}
+#endif
+
+// Clear all messages (RAM + persisted queue)
+void MessageStore::clearAllMessages()
+{
+    std::deque<StoredMessage>().swap(liveMessages);
+    resetMessagePool();
+
+#ifdef FSCom
+    SafeFile f(filename.c_str(), false);
+    uint8_t count = 0;
+    f.write(&count, 1); // write "0 messages"
+    f.close();
+#endif
+}
+
+// Internal helper: erase first or last message matching a predicate
+template <typename Predicate> static void eraseIf(std::deque<StoredMessage> &deque, Predicate pred, bool fromBack = false)
+{
+    if (fromBack) {
+        // Iterate from the back and erase all matches from the end
+        for (auto it = deque.rbegin(); it != deque.rend();) {
+            if (pred(*it)) {
+                it = std::deque<StoredMessage>::reverse_iterator(deque.erase(std::next(it).base()));
+            } else {
+                ++it;
+            }
+        }
+    } else {
+        // Manual forward search to erase all matches
+        for (auto it = deque.begin(); it != deque.end();) {
+            if (pred(*it)) {
+                it = deque.erase(it);
+            } else {
+                ++it;
+            }
+        }
+    }
+}
+
+// Delete oldest message (RAM + persisted queue)
+void MessageStore::deleteOldestMessage()
+{
+    eraseIf(liveMessages, [](StoredMessage &) { return true; });
+    saveToFlash();
+}
+
+// Delete oldest message in a specific channel
+void MessageStore::deleteOldestMessageInChannel(uint8_t channel)
+{
+    auto pred = [channel](const StoredMessage &m) { return m.type == MessageType::BROADCAST && m.channelIndex == channel; };
+    eraseIf(liveMessages, pred);
+    saveToFlash();
+}
+
+void MessageStore::deleteAllMessagesInChannel(uint8_t channel)
+{
+    auto pred = [channel](const StoredMessage &m) { return m.type == MessageType::BROADCAST && m.channelIndex == channel; };
+    eraseIf(liveMessages, pred, false /* delete ALL, not just first */);
+    saveToFlash();
+}
+
+void MessageStore::deleteAllMessagesWithPeer(uint32_t peer)
+{
+    uint32_t local = nodeDB->getNodeNum();
+    auto pred = [&](const StoredMessage &m) {
+        if (m.type != MessageType::DM_TO_US)
+            return false;
+        uint32_t other = (m.sender == local) ? m.dest : m.sender;
+        return other == peer;
+    };
+    eraseIf(liveMessages, pred, false);
+    saveToFlash();
+}
+
+// Delete oldest message in a direct chat with a node
+void MessageStore::deleteOldestMessageWithPeer(uint32_t peer)
+{
+    auto pred = [peer](const StoredMessage &m) {
+        if (m.type != MessageType::DM_TO_US)
+            return false;
+        uint32_t other = (m.sender == nodeDB->getNodeNum()) ? m.dest : m.sender;
+        return other == peer;
+    };
+    eraseIf(liveMessages, pred);
+    saveToFlash();
+}
+
+std::deque<StoredMessage> MessageStore::getChannelMessages(uint8_t channel) const
+{
+    std::deque<StoredMessage> result;
+    for (const auto &m : liveMessages) {
+        if (m.type == MessageType::BROADCAST && m.channelIndex == channel) {
+            result.push_back(m);
+        }
+    }
+    return result;
+}
+
+std::deque<StoredMessage> MessageStore::getDirectMessages() const
+{
+    std::deque<StoredMessage> result;
+    for (const auto &m : liveMessages) {
+        if (m.type == MessageType::DM_TO_US) {
+            result.push_back(m);
+        }
+    }
+    return result;
+}
+
+// Upgrade boot-relative timestamps once RTC is valid
+// Only same-boot boot-relative messages are healed.
+// Persisted boot-relative messages from old boots stay ??? forever.
+void MessageStore::upgradeBootRelativeTimestamps()
+{
+    uint32_t nowSecs = getValidTime(RTCQuality::RTCQualityDevice, true);
+    if (nowSecs == 0)
+        return; // Still no valid RTC
+
+    uint32_t bootNow = millis() / 1000;
+
+    auto fix = [&](std::deque<StoredMessage> &dq) {
+        for (auto &m : dq) {
+            if (m.isBootRelative && m.timestamp <= bootNow) {
+                uint32_t bootOffset = nowSecs - bootNow;
+                m.timestamp += bootOffset;
+                m.isBootRelative = false;
+            }
+        }
+    };
+    fix(liveMessages);
+}
+
+const char *MessageStore::getText(const StoredMessage &msg)
+{
+    // Wrapper around the internal helper
+    return getTextFromPool(msg.textOffset);
+}
+
+uint16_t MessageStore::storeText(const char *src, size_t len)
+{
+    // Wrapper around the internal helper
+    return storeTextInPool(src, len);
+}
+
+// Global definition
+MessageStore messageStore("default");
+#endif